Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-Modal Time Series Prediction via Mixture of Modulated Experts

Lige Zhang, Ali Maatouk, Jialin Chen, Leandros Tassiulas, Rex Ying

TL;DR

The paper tackles the challenge of multi-modal time-series forecasting when text signals supplement temporal data. It replaces conventional token-level fusion with Expert Modulation (MoME), a cross-modal mechanism that conditions both routing and per-expert computation on textual context derived from large language models. The authors provide a geometric interpretation of MoE, present a three-step EM framework (Context Token Distillation, Router Modulation, EiLM), and demonstrate backbone-agnostic improvements across diverse datasets and baselines. Empirical results show MoME achieves consistent gains over uni-modal and token-fusion approaches while offering improved training efficiency and robustness to noisy text. The work suggests a scalable path for integrating auxiliary modalities into time-series models and points to future extensions to additional modalities and bidirectional modulation with LLMs.

Abstract

Real-world time series exhibit complex and evolving dynamics, making accurate forecasting extremely challenging. Recent multi-modal forecasting methods leverage textual information such as news reports to improve prediction, but most rely on token-level fusion that mixes temporal patches with language tokens in a shared embedding space. However, such fusion can be ill-suited when high-quality time-text pairs are scarce and when time series exhibit substantial variation in scale and characteristics, thus complicating cross-modal alignment. In parallel, Mixture-of-Experts (MoE) architectures have proven effective for both time series modeling and multi-modal learning, yet many existing MoE-based modality integration methods still depend on token-level fusion. To address this, we propose Expert Modulation, a new paradigm for multi-modal time series prediction that conditions both routing and expert computation on textual signals, enabling direct and efficient cross-modal control over expert behavior. Through comprehensive theoretical analysis and experiments, our proposed method demonstrates substantial improvements in multi-modal time series prediction. The current code is available at https://github.com/BruceZhangReve/MoME

Multi-Modal Time Series Prediction via Mixture of Modulated Experts

TL;DR

The paper tackles the challenge of multi-modal time-series forecasting when text signals supplement temporal data. It replaces conventional token-level fusion with Expert Modulation (MoME), a cross-modal mechanism that conditions both routing and per-expert computation on textual context derived from large language models. The authors provide a geometric interpretation of MoE, present a three-step EM framework (Context Token Distillation, Router Modulation, EiLM), and demonstrate backbone-agnostic improvements across diverse datasets and baselines. Empirical results show MoME achieves consistent gains over uni-modal and token-fusion approaches while offering improved training efficiency and robustness to noisy text. The work suggests a scalable path for integrating auxiliary modalities into time-series models and points to future extensions to additional modalities and bidirectional modulation with LLMs.

Abstract

Real-world time series exhibit complex and evolving dynamics, making accurate forecasting extremely challenging. Recent multi-modal forecasting methods leverage textual information such as news reports to improve prediction, but most rely on token-level fusion that mixes temporal patches with language tokens in a shared embedding space. However, such fusion can be ill-suited when high-quality time-text pairs are scarce and when time series exhibit substantial variation in scale and characteristics, thus complicating cross-modal alignment. In parallel, Mixture-of-Experts (MoE) architectures have proven effective for both time series modeling and multi-modal learning, yet many existing MoE-based modality integration methods still depend on token-level fusion. To address this, we propose Expert Modulation, a new paradigm for multi-modal time series prediction that conditions both routing and expert computation on textual signals, enabling direct and efficient cross-modal control over expert behavior. Through comprehensive theoretical analysis and experiments, our proposed method demonstrates substantial improvements in multi-modal time series prediction. The current code is available at https://github.com/BruceZhangReve/MoME
Paper Structure (62 sections, 2 theorems, 41 equations, 12 figures, 12 tables, 5 algorithms)

This paper contains 62 sections, 2 theorems, 41 equations, 12 figures, 12 tables, 5 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. Time Series Prediction via MoE.
  4. Multi-Modal Time Series Prediction.
  5. Multi-Modal Learning via MoE.
  6. Preliminary
  7. Multi-Modal Time Series
  8. Mixture of Experts
  9. Mixture of Modulated Experts
  10. MoE from a Geometric Perspective
  11. Cross-Modal Interaction via Expert Modulation
  12. Step 1: Context Token Distillation.
  13. Step 2: Router Modulation.
  14. Step 3: Expert independent Linear Modulation.
  15. Complexity Analysis
  16. ...and 47 more sections

Key Result

Lemma 4.1

Let $f(\cdot)$ be a single layer MLP with $E \times C$ hidden units. Then, $f(\cdot)$ can be equivalently written as $f(\mathbf{x}) = \sum_{i=1}^{E} f_i(\mathbf{x})$, where each $f_i(\cdot)$ is a single-layer MLP with $C$ hidden units, corresponding to a disjoint group of hidden neurons.

Figures (12)

  • Figure 1: A large language model generates modulation signals that condition both routing and expert computation in a time series MoE model, enabling cross-modal control over temporal experts.
  • Figure 2: (a): classical Token-level fusion approach, where Fuse(1) and Fuse(2) denote potential stages that cross-modal fusion to be performed. (b): an instantiation of the proposed Expert Modulation multi-modal learning approach. It consists of two main components: EiLM (Expert independent Linear Modulation) and RM (Router Modulation).
  • Figure 3: (a): A case study comparison between the Expert Modulation (EM) based multi-modal MoME and the uni-modal variant MoME ($w/o$ EM). (b): The change in routing behavior when activating router modulation ($w/o$ RM).
  • Figure 4: (a): Comparison between token fusion and expert modulation. (b): Comparison on hyperparameter effects.
  • Figure 5: (a): empirical results that show sparse expert activation yields better performance than dense activation; (b): expert selection behavior for MoE-based model on ETTm2 datasets, where a channel is considered as a token.
  • ...and 7 more figures

Theorems & Definitions (2)

  • Lemma 4.1: Decomposition of MLP
  • Theorem 4.2: Error Bound of Sparse MoE