MDMLP-EIA: Multi-domain Dynamic MLPs with Energy Invariant Attention for Time Series Forecasting

TL;DR

An adaptive fused dual-domain seasonal MLP that categorizes seasonal signals into strong and weak components and an energy invariant attention mechanism that adaptively focuses on different feature channels within trend and seasonal predictions across time steps is developed.

Abstract

Time series forecasting is essential across diverse domains. While MLP-based methods have gained attention for achieving Transformer-comparable performance with fewer parameters and better robustness, they face critical limitations including loss of weak seasonal signals, capacity constraints in weight-sharing MLPs, and insufficient channel fusion in channel-independent strategies. To address these challenges, we propose MDMLP-EIA (Multi-domain Dynamic MLPs with Energy Invariant Attention) with three key innovations. First, we develop an adaptive fused dual-domain seasonal MLP that categorizes seasonal signals into strong and weak components. It employs an adaptive zero-initialized channel fusion strategy to minimize noise interference while effectively integrating predictions. Second, we introduce an energy invariant attention mechanism that adaptively focuses on different feature channels within trend and seasonal predictions across time steps. This mechanism maintains constant total signal energy to align with the decomposition-prediction-reconstruction framework and enhance robustness against disturbances. Third, we propose a dynamic capacity adjustment mechanism for channel-independent MLPs. This mechanism scales neuron count with the square root of channel count, ensuring sufficient capacity as channels increase. Extensive experiments across nine benchmark datasets demonstrate that MDMLP-EIA achieves state-of-the-art performance in both prediction accuracy and computational efficiency.

Figures (13)

• Figure 1: Limitations of current MLP-based methods. (a) Loss of weak seasonal signals: Weak seasonal signals closely resemble noise signals and are difficult to distinguish from them. When applying frequency domain amplitude restrictions to reduce noise, some weak seasonal signals are inevitably lost. (b) Different capacity requirements of channel-independent MLPs for prediction tasks with varying numbers of channels: For prediction tasks with larger numbers of channels (such as Traffic), MLPs require greater capacity to meet the predictive demands of each channel. Conversely, for prediction tasks with fewer channels (such as ETTh1), MLPs require smaller capacity to prevent model overfitting. (c) Reduced Mean Squared Errors (MSEs) achieved by the proposed energy invariant attention at varied prediction lengths in some datasets.
• Figure 2: MDMLP-EIA overall architecture. (i) RevIN normalization and EMA decompose input series into trend and seasonal components; (ii) Trend component feeds into trend MLP; seasonal component processes through our adaptive fused dual-domain seasonal MLP with adaptive zero-initialized channel fusion); (iii) Our energy invariant attention module merges trend and seasonal predictions; (iv) Our Dynamic capacity adjustment mechanism optimizes multi-domain MLPs.
• Figure 3: Model effectiveness and efficiency comparison on the Exchange and Electricity datasets.
• Figure 4: cof results under different C and $\tau$ values.
• Figure 5: MDMLP-EIA (a) demonstrates enhanced prediction accuracy on the Weather dataset ($L=96, T=192$), more closely tracking complex dynamics and key turning points of the ground truth than baseline models Amplifier (b), xPatch (c), and iTransformer (d).

Theorems & Definitions (3)

• Proposition E.1: Strict error reduction and optimal initialization
• Theorem F.1: Non-inferiority of energy-invariant attention
• proof