Dual-Forecaster: A Multimodal Time Series Model Integrating Descriptive and Predictive Texts

TL;DR

Dual-Forecaster tackles information scarcity in time-series forecasting by fusing descriptive historical text and predictive textual cues with numerical series. It employs a dual-branch architecture (RoBERTa-based textual branch and PaLM-backed temporal branch) and three cross-modality alignment mechanisms to learn robust text–time relationships, with a forecast head that outputs Student's $T$ distribution parameters. The training objective combines ${\mathcal{L}}_{forecast}$ and ${\mathcal{L}}_{contrastive}$, formalized as ${\mathcal{L}}={\mathcal{L}}_{forecast}+{\mathcal{L}}_{contrastive}$, and forecasts are produced via $p(\mathbf{X}_{t:t+h} \mid \hat{\boldsymbol{\phi}})$ using the last token embedding. Across fifteen multimodal datasets, including synthetic, captioned-public, and real multimodal sets, the approach consistently matches or surpasses state-of-the-art baselines, with ablations confirming the complementary value of descriptively historical and predictive texts for forecasting.

Abstract

Most existing single-modal time series models rely solely on numerical series, which suffer from the limitations imposed by insufficient information. Recent studies have revealed that multimodal models can address the core issue by integrating textual information. However, these models focus on either historical or future textual information, overlooking the unique contributions each plays in time series forecasting. Besides, these models fail to grasp the intricate relationships between textual and time series data, constrained by their moderate capacity for multimodal comprehension. To tackle these challenges, we propose Dual-Forecaster, a pioneering multimodal time series model that combines both descriptively historical textual information and predictive textual insights, leveraging advanced multimodal comprehension capability empowered by three well-designed cross-modality alignment techniques. Our comprehensive evaluations on fifteen multimodal time series datasets demonstrate that Dual-Forecaster is a distinctly effective multimodal time series model that outperforms or is comparable to other state-of-the-art models, highlighting the superiority of integrating textual information for time series forecasting. This work opens new avenues in the integration of textual information with numerical time series data for multimodal time series analysis.

Figures (16)

• Figure 1: Overall architecture of Dual-Forecaster. Top left is the text branch with text input, and top right is the temporal branch with time series input. Based on the obtained text features and time series embeddings, to improve the model's multimodal comprehension capability, we employ three cross-modality alignment techniques: historical text-time series contrastive loss and history-oriented modality interaction module (Modality Interaction I), as well as future-oriented modality interaction module (Modality Interaction II). The outputs of time series embeddings from Modality Interaction II are projected to generate the final forecasts.
• Figure 2: Visualization of an example from synthetic dataset under the input-100-predict-30 settings.
• Figure 3: A showcase of text-time series alignment. The values in the matrix represent the similarity between the high-dimensional representation of the time series (above the matrix) and the corresponding textual description (on the left side of the matrix). The higher the similarity, the better the match between the time series and the text.
• Figure 4: Synthetic time series and its paired text examples.
• Figure 5: Captioning process for real-world time series. First, IEPF is used to segment time series, identifying reasonable segmentation points. This algorithm works by iteratively fitting straight lines between endpoints and adjusting segmentation points to minimize fitting errors, thereby identifying rational breakpoints. Next, statistical features such as slope and volatility are calculated for each segmented portion of the time series. Finally, based on these statistical characteristics, a descriptive textual summary is generated.