FreqMoE: Enhancing Time Series Forecasting through Frequency Decomposition Mixture of Experts

TL;DR

FreqMoE tackles long-horizon time-series forecasting by leveraging frequency-domain representations and dynamic, learnable frequency-band decomposition. It introduces a frequency decomposition mixture-of-experts module with learnable boundaries and a gating network that allocates emphasis across bands, followed by a residual-connected complex-valued prediction stack that refines forecasts in the frequency domain before returning to the time domain. The approach yields state-of-the-art results on numerous benchmarks while maintaining a fraction of the parameters of Transformer-based rivals, and ablations confirm the necessity of the gating mechanism and the appropriate number of experts. The work demonstrates that preserving and adaptively weighting information across frequency bands enhances long-range pattern capture and offers practical efficiency advantages for real-world forecasting tasks.

Abstract

Long-term time series forecasting is essential in areas like finance and weather prediction. Besides traditional methods that operate in the time domain, many recent models transform time series data into the frequency domain to better capture complex patterns. However, these methods often use filtering techniques to remove certain frequency signals as noise, which may unintentionally discard important information and reduce prediction accuracy. To address this, we propose the Frequency Decomposition Mixture-of-Experts (FreqMoE) model, which dynamically decomposes time series data into frequency bands, each processed by a specialized expert. A gating mechanism adjusts the importance of each output of expert based on frequency characteristics, and the aggregated results are fed into a prediction module that iteratively refines the forecast using residual connections. Our experiments demonstrate that FreqMoE outperforms state-of-the-art models, achieving the best performance on 51 out of 70 metrics across all tested datasets, while significantly reducing the number of required parameters to under 50k, providing notable efficiency advantages. Code is available at: https://github.com/sunbus100/FreqMoE-main

Figures (5)

• Figure 1: The complete structure of the FreqMoE model consists of a Frequency Decomposition Mixture of Experts Block, which decomposes and processes the input time series before reconstruction, and a Frequency Domain Prediction Stack, which is composed of multiple prediction blocks connected through residual connections. This model dynamically captures patterns in the sequence by leveraging the relationships and frequency characteristics across different frequency bands of the input data, enabling accurate predictions.
• Figure 2: Heatmap representation of gating coefficient sequences across different frequency bands. The X-axis denotes the frequency bands (experts), and the Y-axis represents the sequence indices. The color intensity in each cell corresponds to the gating coefficient, with higher values indicating a greater contribution from the respective frequency band to the prediction of the input sequence.
• Figure 3: The subplot on the left shows the loss for models with different numbers of experts, and the subplot on the right shows the distribution of the bandwidths of the models with different numbers of experts, where the widths are the relative widths of the bands with respect to the sequence.
• Figure 4: This figure illustrates the average coefficient assigned to each expert by the gating network, along with the bandwidth managed by each expert.
• Figure 5: Visualization of the synthetic dataset and expert weight. In heatmaps, positive values are shown with bright colors, while negative values are represented by darker shades.