Seg-MoE: Multi-Resolution Segment-wise Mixture-of-Experts for Time Series Forecasting Transformers

TL;DR

Seg-MoE tackles the scalability and locality challenges of Transformer-based time-series forecasters by introducing segment-wise routing for Mixture-of-Experts. By partitioning the input into contiguous time-step segments and routing each segment to a small subset of experts (plus a shared fallback), the model captures intra-segment interactions and local temporal patterns more effectively than token-wise MoE. The approach is augmented with a multi-resolution routing scheme, a robust loss with routing balance, and an encoder-only Transformer backbone featuring patch embeddings, channel independence, and RoPE with efficient attention. Empirically, Seg-MoE achieves state-of-the-art performance on seven multivariate forecasting benchmarks across horizons up to $H=720$, with notable gains at long horizons and ablations confirming the critical role of segment-level routing and temporal hierarchy. These findings suggest segment-aligned inductive biases can substantially improve accuracy and efficiency for sequential data modeling, and point to promising directions like adaptive segment lengths and diverse expert architectures for future large-scale time-series models.

Abstract

Transformer-based models have recently made significant advances in accurate time-series forecasting, but even these architectures struggle to scale efficiently while capturing long-term temporal dynamics. Mixture-of-Experts (MoE) layers are a proven solution to scaling problems in natural language processing. However, existing MoE approaches for time-series forecasting rely on token-wise routing mechanisms, which may fail to exploit the natural locality and continuity of temporal data. In this work, we introduce Seg-MoE, a sparse MoE design that routes and processes contiguous time-step segments rather than making independent expert decisions. Token segments allow each expert to model intra-segment interactions directly, naturally aligning with inherent temporal patterns. We integrate Seg-MoE layers into a time-series Transformer and evaluate it on multiple multivariate long-term forecasting benchmarks. Seg-MoE consistently achieves state-of-the-art forecasting accuracy across almost all prediction horizons, outperforming both dense Transformers and prior token-wise MoE models. Comprehensive ablation studies confirm that segment-level routing is the key factor driving these gains. Our results show that aligning the MoE routing granularity with the inherent structure of time series provides a powerful, yet previously underexplored, inductive bias, opening new avenues for conditionally sparse architectures in sequential data modeling.

Figures (10)

• Figure 1: Mixture-of-Experts ($\operatorname{MoE}$) designs for sparse conditional computation in Transformer blocks. (a) Standard token-wise $\operatorname{MoE}$: a router computes token-to-expert affinities and selects $\operatorname{Top-K}$ routed experts from $N$ experts; the layer output is the weighted sum of the selected expert outputs. (b) Seg-MoE: routing is performed at the segment level, and the output combines $\operatorname{Top-K}$ routed experts with an always-active shared expert, providing a stable, dense pathway while preserving sparsity in the routed experts.
• Figure 2: Encoder-only Transformer architecture used to experiment with Seg-MoE layers in time series forecasting.
• Figure 3: Memory footprint during training sweeping the segment resolution, where $\omega=1$ is equivalent to a standard $\operatorname{MoE}$ layer, and $\omega>1$ corresponds to our Seg-MoE.
• Figure 4: Forecast showcases from ETTh1 with a forecast horizon of 96. Blue curves are the ground truths, and orange curves are the model predictions. The curves before the model predictions are the input data.
• Figure 5: Forecast showcases from ETTh2 with a forecast horizon of 96. Blue curves are the ground truths, and orange curves are the model predictions. The curves before the model predictions are the input data.