Spectral Text Fusion: A Frequency-Aware Approach to Multimodal Time-Series Forecasting

TL;DR

SpecTF tackles multimodal time-series forecasting by modeling textual influence in the frequency domain. It decomposes time series into spectral components and fuses them with text via a cross-modal attention mechanism (FreqCMF), reweighting frequency bands before reconstructing time-domain forecasts. Empirical results on Time-MMD and TimeText Corpus show SpecTF achieving state-of-the-art or competitive performance with substantially fewer parameters than competing methods. The approach foregrounds multiscale text–time interactions and offers practical efficiency for large-scale deployment, with potential extensions toward explainability and fairness in spectral-text alignment.

Abstract

Multimodal time series forecasting is crucial in real-world applications, where decisions depend on both numerical data and contextual signals. The core challenge is to effectively combine temporal numerical patterns with the context embedded in other modalities, such as text. While most existing methods align textual features with time-series patterns one step at a time, they neglect the multiscale temporal influences of contextual information such as time-series cycles and dynamic shifts. This mismatch between local alignment and global textual context can be addressed by spectral decomposition, which separates time series into frequency components capturing both short-term changes and long-term trends. In this paper, we propose SpecTF, a simple yet effective framework that integrates the effect of textual data on time series in the frequency domain. Our method extracts textual embeddings, projects them into the frequency domain, and fuses them with the time series' spectral components using a lightweight cross-attention mechanism. This adaptively reweights frequency bands based on textual relevance before mapping the results back to the temporal domain for predictions. Experimental results demonstrate that SpecTF significantly outperforms state-of-the-art models across diverse multi-modal time series datasets while utilizing considerably fewer parameters. Code is available at https://github.com/hiepnh137/SpecTF.

Figures (6)

• Figure 1: An example of SpecTF's predictions in Agriculture data. We provide both time-domain comparison (a) and frequency-domain comparison (b) in the predictions when incorporating text, compared to when it is not used.
• Figure 2: An overview of the proposed SpecTF framework. The time series input is passed through Time Series Embedding to be encoded in the frequency domain, and concurrently, Text Embedding represents each document in the frequency domain. Frequency Cross-Modality Fusion (FreqCMF) integrates textual information into time series frequencies through attention mechanisms and multiplication fusion based on complex multiplication operation $\odot$. Forecaster that maps historical frequency representations to prediction representations in the frequency domain. Projection projects the representations to the original dimension of the spectrum and converts them back to the temporal domain.
• Figure 3: Visualization of frequency-text interactions in the Climate dataset. The attention map (center) reveals document-frequency relationships, with brightness indicating attention strength. Documents 13, 16, and 20 (left panel) show strong attention weights in middle-frequency bands (2-7), highlighted by the red rectangle. The frequency amplitude comparison (right) confirms this relationship: amplitudes after fusion (orange) show significant increases in these middle frequencies compared to the values before fusion (blue), as indicated by the corresponding red rectangle.
• Figure 4: Parameter study on learning rate.
• Figure 5: Parameter study on model dimension $d$.