Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Rethinking Multimodal Fusion for Time Series: Auxiliary Modalities Need Constrained Fusion

Seunghan Lee, Jun Seo, Jaehoon Lee, Sungdong Yoo, Minjae Kim, Tae Yoon Lim, Dongwan Kang, Hwanil Choi, SoonYoung Lee, Wonbin Ahn

Abstract

Recent advances in multimodal learning have motivated the integration of auxiliary modalities such as text or vision into time series (TS) forecasting. However, most existing methods provide limited gains, often improving performance only in specific datasets or relying on architecture-specific designs that limit generalization. In this paper, we show that multimodal models with naive fusion strategies (e.g., simple addition or concatenation) often underperform unimodal TS models, which we attribute to the uncontrolled integration of auxiliary modalities which may introduce irrelevant information. Motivated by this observation, we explore various constrained fusion methods designed to control such integration and find that they consistently outperform naive fusion methods. Furthermore, we propose Controlled Fusion Adapter (CFA), a simple plug-in method that enables controlled cross-modal interactions without modifying the TS backbone, integrating only relevant textual information aligned with TS dynamics. CFA employs low-rank adapters to filter irrelevant textual information before fusing it into temporal representations. We conduct over 20K experiments across various datasets and TS/text models, demonstrating the effectiveness of the constrained fusion methods including CFA. Code is publicly available at: https://github.com/seunghan96/cfa/.

Rethinking Multimodal Fusion for Time Series: Auxiliary Modalities Need Constrained Fusion

Abstract

Recent advances in multimodal learning have motivated the integration of auxiliary modalities such as text or vision into time series (TS) forecasting. However, most existing methods provide limited gains, often improving performance only in specific datasets or relying on architecture-specific designs that limit generalization. In this paper, we show that multimodal models with naive fusion strategies (e.g., simple addition or concatenation) often underperform unimodal TS models, which we attribute to the uncontrolled integration of auxiliary modalities which may introduce irrelevant information. Motivated by this observation, we explore various constrained fusion methods designed to control such integration and find that they consistently outperform naive fusion methods. Furthermore, we propose Controlled Fusion Adapter (CFA), a simple plug-in method that enables controlled cross-modal interactions without modifying the TS backbone, integrating only relevant textual information aligned with TS dynamics. CFA employs low-rank adapters to filter irrelevant textual information before fusing it into temporal representations. We conduct over 20K experiments across various datasets and TS/text models, demonstrating the effectiveness of the constrained fusion methods including CFA. Code is publicly available at: https://github.com/seunghan96/cfa/.
Paper Structure (35 sections, 17 equations, 14 figures, 72 tables, 1 algorithm)

This paper contains 35 sections, 17 equations, 14 figures, 72 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Necessity of Constrained Fusion for Multimodal TS Forecasting
  4. Naive Fusion
  5. Constrained Fusion
  6. Controlled Fusion Adapter (CFA)
  7. Experiments
  8. Multimodal Time Series Forecasting
  9. Performance with Various Settings
  10. Analysis
  11. Conclusion
  12. Details of Models
  13. Details of Datasets
  14. Experimental Setup
  15. Model Hyperparameters
  16. ...and 20 more sections

Figures (14)

  • Figure 1: Constrained vs. naive fusion.
  • Figure 2: Comparison of multimodal fusion strategies for TS. (Left) Naive fusion applies simple additive or concatenation operators at first, middle, or last stages without considering modality relevance. (Right) Constrained fusion incorporates textual information in a controlled manner by considering its relevance to TS. CFA injects textual signals via a residual connection constrained to a low-rank subspace to filter irrelevant information while preserving TS representations.
  • Figure 3: Comparison of constrained fusions.
  • Figure 4: Performance across diverse settings. (a), (b), and (c) show the performance (normalized MSE) by Dataset, TS model, and Text model, respectively. (d) shows the overall average performance across all settings. CFA ($\star$) consistently achieves the lowest MSE among fusion strategies.
  • Figure 5: Irrelevant text experiments.
  • ...and 9 more figures