Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MGTS-Net: Exploring Graph-Enhanced Multimodal Fusion for Augmented Time Series Forecasting

Shule Hao, Junpeng Bao, Wenli Li

TL;DR

The paper tackles time series forecasting with multimodal data by integrating text, images, and series within a graph-enhanced framework to capture fine-grained temporal patterns and multi-scale dynamics. MGTS-Net introduces the Multimodal Feature Extraction Layer (MFE) with a Frequency-Time Cell (FTC) integrated TIME-MOE, the Multimodal Feature Fusion Layer (MFF) as a heterogeneous graph with cross-modal and intra-modal relations, and the Multi-Scale Prediction Layer (MSP) with three heads and adaptive fusion (weights $w$ computed by an MLP over input features). Experiments on standard benchmarks show MGTS-Net achieves superior accuracy and efficiency, including strong performance in few-shot settings, with ablations confirming the necessity of each component. The approach enhances cross-modal alignment and dynamic multi-scale fusion, offering a scalable and practical solution for real-world forecasting where auxiliary data may be scarce.

Abstract

Recent research in time series forecasting has explored integrating multimodal features into models to improve accuracy. However, the accuracy of such methods is constrained by three key challenges: inadequate extraction of fine-grained temporal patterns, suboptimal integration of multimodal information, and limited adaptability to dynamic multi-scale features. To address these problems, we propose MGTS-Net, a Multimodal Graph-enhanced Network for Time Series forecasting. The model consists of three core components: (1) a Multimodal Feature Extraction layer (MFE), which optimizes feature encoders according to the characteristics of temporal, visual, and textual modalities to extract temporal features of fine-grained patterns; (2) a Multimodal Feature Fusion layer (MFF), which constructs a heterogeneous graph to model intra-modal temporal dependencies and cross-modal alignment relationships and dynamically aggregates multimodal knowledge; (3) a Multi-Scale Prediction layer (MSP), which adapts to multi-scale features by dynamically weighting and fusing the outputs of short-term, medium-term, and long-term predictors. Extensive experiments demonstrate that MGTS-Net exhibits excellent performance with light weight and high efficiency. Compared with other state-of-the-art baseline models, our method achieves superior performance, validating the superiority of the proposed methodology.

MGTS-Net: Exploring Graph-Enhanced Multimodal Fusion for Augmented Time Series Forecasting

TL;DR

The paper tackles time series forecasting with multimodal data by integrating text, images, and series within a graph-enhanced framework to capture fine-grained temporal patterns and multi-scale dynamics. MGTS-Net introduces the Multimodal Feature Extraction Layer (MFE) with a Frequency-Time Cell (FTC) integrated TIME-MOE, the Multimodal Feature Fusion Layer (MFF) as a heterogeneous graph with cross-modal and intra-modal relations, and the Multi-Scale Prediction Layer (MSP) with three heads and adaptive fusion (weights computed by an MLP over input features). Experiments on standard benchmarks show MGTS-Net achieves superior accuracy and efficiency, including strong performance in few-shot settings, with ablations confirming the necessity of each component. The approach enhances cross-modal alignment and dynamic multi-scale fusion, offering a scalable and practical solution for real-world forecasting where auxiliary data may be scarce.

Abstract

Recent research in time series forecasting has explored integrating multimodal features into models to improve accuracy. However, the accuracy of such methods is constrained by three key challenges: inadequate extraction of fine-grained temporal patterns, suboptimal integration of multimodal information, and limited adaptability to dynamic multi-scale features. To address these problems, we propose MGTS-Net, a Multimodal Graph-enhanced Network for Time Series forecasting. The model consists of three core components: (1) a Multimodal Feature Extraction layer (MFE), which optimizes feature encoders according to the characteristics of temporal, visual, and textual modalities to extract temporal features of fine-grained patterns; (2) a Multimodal Feature Fusion layer (MFF), which constructs a heterogeneous graph to model intra-modal temporal dependencies and cross-modal alignment relationships and dynamically aggregates multimodal knowledge; (3) a Multi-Scale Prediction layer (MSP), which adapts to multi-scale features by dynamically weighting and fusing the outputs of short-term, medium-term, and long-term predictors. Extensive experiments demonstrate that MGTS-Net exhibits excellent performance with light weight and high efficiency. Compared with other state-of-the-art baseline models, our method achieves superior performance, validating the superiority of the proposed methodology.
Paper Structure (30 sections, 16 equations, 4 figures, 4 tables, 1 algorithm)

This paper contains 30 sections, 16 equations, 4 figures, 4 tables, 1 algorithm.

Table of Contents

  1. 1.Introduction
  2. 2. Related Work
  3. 2.1 Unimodal Time Series Forecasting
  4. 2.2 Multimodal Time Series Forecasting
  5. 2.3 Graph-Enhanced Time Series Models
  6. 3. Methodology
  7. 3.1 Multimodal Feature Extraction Layer
  8. Text Feature Extraction Module
  9. Temporal Feature Extraction Module
  10. Image Feature Extraction Module
  11. 3.2 Multimodal Feature Fusion layer
  12. Multimodal Relations
  13. Past Relations
  14. Future Relations
  15. 3.3 Multi-Scale Prediction layer
  16. ...and 15 more sections

Figures (4)

  • Figure 1: Overview of the MGTS-Net framework.
  • Figure 2: Results of averaging the parameter results of different prediction heads over the forcasting horizons $H \in \{96, 192, 336, 720\}$
  • Figure 3: Visual comparison of image feature extraction between MGTS-Net and VIT: The first seven blocks represent the features of each image patch, and the last block shows the features of 14x14 patches in the VIT model.
  • Figure 4: Interpretability visualization of MGTS-Net: multimodal feature alignment via UMAP.