Multimodal Conditioned Diffusive Time Series Forecasting

TL;DR

The paper addresses the challenge of forecasting time series by leveraging multimodal information. It introduces MCD-TSF, a diffusion-based framework that conditions forecasts on historical series $\mathcal{X}$, timestamps $\mathcal{U}$, and textual descriptions $\mathcal{E}$, implemented through a $K$-step denoising process and guided by classifier-free guidance to balance modalities. Key contributions include (i) a multimodal encoder, (ii) Transformer-based fusion layers with Timestamp-Assisted Attention (TAA) and Text-Time Fusion (TTF), and (iii) an adaptive output fusion mechanism that combines timestamp- and text-conditioned predictions, achieving state-of-the-art performance across eight real-world domains. The results demonstrate improved predictive accuracy and robustness to textual noise, offering uncertainty-aware forecasts and scalable multimodal TSF with practical impact across diverse sectors.

Abstract

Diffusion models achieve remarkable success in processing images and text, and have been extended to special domains such as time series forecasting (TSF). Existing diffusion-based approaches for TSF primarily focus on modeling single-modality numerical sequences, overlooking the rich multimodal information in time series data. To effectively leverage such information for prediction, we propose a multimodal conditioned diffusion model for TSF, namely, MCD-TSF, to jointly utilize timestamps and texts as extra guidance for time series modeling, especially for forecasting. Specifically, Timestamps are combined with time series to establish temporal and semantic correlations among different data points when aggregating information along the temporal dimension. Texts serve as supplementary descriptions of time series' history, and adaptively aligned with data points as well as dynamically controlled in a classifier-free manner. Extensive experiments on real-world benchmark datasets across eight domains demonstrate that the proposed MCD-TSF model achieves state-of-the-art performance.

Figures (7)

• Figure 1: The overall architecture of the proposed MCD-TSF. Our approach consists of four components: the diffusion framework; a multimodal encoder, multimodal fusion module, and an output layer, which are presented at the top, bottom-left, bottom-center, and bottom-right parts of the figure, respectively. The multimodal fusion component internally contains the TAA and TTF modules.
• Figure 2: Results of our approach on different domains when configured with various values of the timestamp weights (i.e., the $\lambda=0.2, 0.4, 0.6, 0.8, 1.0$).
• Figure 3: Visual of the attention of our approach across data points of two cases with different settings, where darker colors refer to higher attention weights. Figures (a) and (c) are the attentions from models without using timestamps for case 1 and case 2, respectively; and Figures (b) and (d) are the attentions from models with timestamps for case 1 and case 2, respectively.
• Figure 4: Figure (a) illustrates how the model's MSE varies with different guidance strengths across datasets from multiple domains, where the reported MSE for each configuration is calculated by subtracting the mean MSE across all configurations within the respective domain. Figure (b) displays the model performance with changes in the unconditional training probability.
• Figure 5: Case study with four examples from energy domains. In all cases, the historical time series and the gold standard future time series are marked by blue and dashed grey lines, respectively. The predictions from PatchTST, CSDI, and our approach (i.e., MCD-TSF) are represented by orange, green, and red lines, respectively.