MoHETS: Long-term Time Series Forecasting with Mixture-of-Heterogeneous-Experts

TL;DR

MoHETS tackles long-horizon forecasting for real-world multivariate time series by introducing Mixture-of-Heterogeneous-Experts (MoHE) within an encoder-only Transformer. The architecture combines a shared DwConvFFN for sequence-level continuity with routed FA-FFNs for patch-level periodicities, and it incorporates exogenous covariates through multimodal cross-attention, plus a convolutional patch decoder for parameter efficiency and horizon flexibility. Across seven benchmarks, MoHETS achieves state-of-the-art results, reducing average MSE by about $12\%$ relative to strong baselines and demonstrating robust specialization to heterogeneous temporal dynamics. This approach offers scalable, accurate forecasting with improved robustness to non-stationarity and strong practical impact for long-term decisions in domains like energy, climate, and transportation.

Abstract

Real-world multivariate time series can exhibit intricate multi-scale structures, including global trends, local periodicities, and non-stationary regimes, which makes long-horizon forecasting challenging. Although sparse Mixture-of-Experts (MoE) approaches improve scalability and specialization, they typically rely on homogeneous MLP experts that poorly capture the diverse temporal dynamics of time series data. We address these limitations with MoHETS, an encoder-only Transformer that integrates sparse Mixture-of-Heterogeneous-Experts (MoHE) layers. MoHE routes temporal patches to a small subset of expert networks, combining a shared depthwise-convolution expert for sequence-level continuity with routed Fourier-based experts for patch-level periodic structures. MoHETS further improves robustness to non-stationary dynamics by incorporating exogenous information via cross-attention over covariate patch embeddings. Finally, we replace parameter-heavy linear projection heads with a lightweight convolutional patch decoder, improving parameter efficiency, reducing training instability, and allowing a single model to generalize across arbitrary forecast horizons. We validate across seven multivariate benchmarks and multiple horizons, with MoHETS consistently achieving state-of-the-art performance, reducing the average MSE by $12\%$ compared to strong recent baselines, demonstrating effective heterogeneous specialization for long-term forecasting.

Figures (11)

• Figure 1: Architecture of MoHETS, an encoder-only transformer for multivariate time-series forecasting. (a) The input embedding module splits time channels into sequences of channel-independent patch embeddings. (b) The exogenous embedding module projects, fuses, and patches covariates with the input series to produce aligned exogenous patch embeddings. These patches are processed through $B$ stacked Transformer blocks; each block is composed of self-attention, cross-attention, and a (c) Mixture-of-Heterogeneous-Experts (MoHE), where a shared depthwise-convolution expert maintains sequence continuity and routed Fourier experts resolve local spectral patterns. (d) The patch decoder head projects final embeddings to forecasting horizons.
• Figure 2: The grouped-query attention (GQA) mechanism with Rotary Position Embeddings (RoPE). Single key and value heads are shared for each group of query heads. In MoHETS, we adopt a grouping factor of two query heads for each key/value head, i.e., Q-heads $= 2 \times$KV-heads.
• Figure 3: Training and validation loss curves of MoHETS on Traffic data, comparing a conventional MLP-based projection head (left) with our convolutional head (right). We combine the Huber loss with the balanced loss for training (see Section \ref{['subsec:objective']}) and use the MSE loss for validation.
• Figure 4: Scalability analysis on ETTm1, ETTm2, Weather, and ECL, with varying $d_{\text{model}}$ sizes on the x-axis. Lower MSE or MAE indicates better performance.
• Figure 5: Forecast showcases of MoHETS across different time channels from ETTh1, with a horizon of 96. Blue curves are the ground truths, and orange curves are the model predictions. Curves before the model predictions are the input data.