MoDEx: Mixture of Depth-specific Experts for Multivariate Long-term Time Series Forecasting

TL;DR

This work addresses the efficiency gap in multivariate long-term time series forecasting by first introducing layer sensitivity $S_l(x)$, a gradient-based metric that reveals depth-specific specialization in backbone layers. It then proposes MoDEx, a Mixture of Depth-specific Experts composed of three depth-varied MLPs, with a residual fusion and a learnable feature translation, achieving state-of-the-art accuracy across seven real-world benchmarks while using far fewer parameters and lower FLOPs. Importantly, the MoDEx module can serve as a drop-in replacement for self-attention in Transformer-based models, consistently boosting performance and demonstrating strong generalizability. Overall, MoDEx provides an efficient, flexible, and high-performance framework for LTSF that highlights the value of depth-aware expert mixtures and gradient-based input attribution in model design.

Abstract

Multivariate long-term time series forecasting (LTSF) supports critical applications such as traffic-flow management, solar-power scheduling, and electricity-transformer monitoring. The existing LTSF paradigms follow a three-stage pipeline of embedding, backbone refinement, and long-horizon prediction. However, the behaviors of individual backbone layers remain underexplored. We introduce layer sensitivity, a gradient-based metric inspired by GradCAM and effective receptive field theory, which quantifies both positive and negative contributions of each time point to a layer's latent features. Applying this metric to a three-layer MLP backbone reveals depth-specific specialization in modeling temporal dynamics in the input sequence. Motivated by these insights, we propose MoDEx, a lightweight Mixture of Depth-specific Experts, which replaces complex backbones with depth-specific MLP experts. MoDEx achieves state-of-the-art accuracy on seven real-world benchmarks, ranking first in 78 percent of cases, while using significantly fewer parameters and computational resources. It also integrates seamlessly into transformer variants, consistently boosting their performance and demonstrating robust generalizability as an efficient and high-performance LTSF framework.

Figures (7)

• Figure 1: Let $L\,l$ be the layer sensitivity of a layer $l$. The top row shows the input sequence and the bottom row its sensitivity. Yellow region mark high-contribution inputs, with uniform yellow indicating global attention. The layer exhibiting global sensitivity varies with the sequence’s intrinsic periodicity and trend.
• Figure 2: Distribution of coordinate-wise mean feature values on the ETTm2 dataset. Applying the Learnable Translation (Blue) shifts these mean values overall toward the positive direction compared to the no-translation baseline (Orange).
• Figure 3: Main architecture of MoDEx. (a) The input sequence is linearly embedded. (b) The embedded vectors pass through a learnable transition layer. (c) MoDEx module comprises three depth-specific MLP experts. (d) Expert outputs are weighted by gating coefficients and aggregated to produce the final prediction.
• Figure 4: Comparison of forecasting performance on the Weather dataset: (a) MoDEx (Ours), (b) SOFTS, (c) iTransformer, and (d) PatchTST.
• Figure 5: (a) Memory consumption versus the number of variates, showing MoDEx’s linear scalability and lower memory usage than SOFTS and iTransformer. (b) and (c) bubble plots on the ETTh1 (batch size: 32) and Electricity (batch size: 16) datasets with input/prediction length of 96; bubble size indicates model size. MoDEx achieves a strong trade-off between speed and accuracy on ETTh1, and maintains the second smallest parameter count on Electricity despite a marginal MSE gap.