Language in the Flow of Time: Time-Series-Paired Texts Weaved into a Unified Temporal Narrative

TL;DR

This work proposes a novel framework, Texts as Time Series (TaTS), which considers the time-series-paired texts to be auxiliary variables of the time series and can enhance multimodal predictive performance without modifying model architectures.

Abstract

While many advances in time series models focus exclusively on numerical data, research on multimodal time series, particularly those involving contextual textual information, remains in its infancy. With recent progress in large language models and time series learning, we revisit the integration of paired texts with time series through the Platonic Representation Hypothesis, which posits that representations of different modalities converge to shared spaces. In this context, we identify that time-series-paired texts may naturally exhibit periodic properties that closely mirror those of the original time series. Building on this insight, we propose a novel framework, Texts as Time Series (TaTS), which considers the time-series-paired texts to be auxiliary variables of the time series. TaTS can be plugged into any existing numerical-only time series models and effectively enable them to handle time series data with paired texts. Through extensive experiments on both multimodal time series forecasting and imputation tasks across benchmark datasets with various existing time series models, we demonstrate that TaTS can enhance multimodal predictive performance without modifying model architectures. Our Code is available at https://github.com/iDEA-iSAIL-Lab-UIUC/TaTS.

Figures (14)

• Figure 1: Mean Square Error of modeling frameworks of time series with paired texts. Full results in Appendix \ref{['ap: full radar']}.
• Figure 2: By overlaying the top frequencies of paired texts (vertical dashed lines) onto the amplitude spectrum of the time series, it is observed that the time-series-paired texts exhibit similar periodic properties that closely mirror those of the original time series. We term this phenomenon Chronological Textual Resonance. More Details are provided in Appendix \ref{['ap: detailed frequency analysis']}.
• Figure 3: Texts as Time Series (TaTS) framework. As paired texts may exhibit behaviors similar to accompanying variables in a time series, TaTS transforms the paired texts into auxiliary variables. These variables augment the numerical sequence, forming a unified multimodal sequence that can be seamlessly integrated into any existing time series model.
• Figure 4: Further analysis of our TaTS framework. (a) Learning rate sensitivity: TaTS maintains stable performance across different learning rates (full results in Appendix \ref{['ap: full hyperparameter learning rate']}). (b) Text Projection Dimension sensitivity: TaTS remains robust across varying $d_{\text{mapped}}$ (full results in Appendix \ref{['ap: full hyperparameter text embedding dimension']}). (c) Varying text encoder: TaTS consistently outperforms baselines across different text encoders (full results in Appendix \ref{['ap: full ablation of text encoder']}). (d) Efficiency: TaTS introduces only a minor parameter increase ($\sim1\%$) but significantly improves the performance according to Table \ref{['tab: main forecasting']}.
• Figure 5: While TaTS incurs a slight increase in training time due to augmented auxiliary variables, it significantly improves forecasting. Full results in Appendix \ref{['ap: full efficiency']}.

Theorems & Definitions (4)

• Proposition A.1
• proof
• Proposition A.2
• proof