Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FreEformer: Frequency Enhanced Transformer for Multivariate Time Series Forecasting

Wenzhen Yue, Yong Liu, Xianghua Ying, Bowei Xing, Ruohao Guo, Ji Shi

TL;DR

<3-5 sentence high-level summary> FreEformer tackles multivariate time series forecasting by leveraging a frequency-domain representation of the data, applying a Transformer to frequency-domain tokens to capture cross-variate dependencies. The key novelty is an enhanced attention mechanism that adds a learnable matrix to the attention scores and normalizes rows, increasing representation diversity and improving gradient flow. The approach is theoretically motivated and empirically validated across 18 real-world datasets, achieving state-of-the-art performance and demonstrating that the enhanced attention can also boost other Transformer forecasters as a plug-in. The work presents a simple yet powerful baseline that highlights the benefits of frequency-domain modeling for robust, cross-variable forecasting.</p>

Abstract

This paper presents \textbf{FreEformer}, a simple yet effective model that leverages a \textbf{Fre}quency \textbf{E}nhanced Trans\textbf{former} for multivariate time series forecasting. Our work is based on the assumption that the frequency spectrum provides a global perspective on the composition of series across various frequencies and is highly suitable for robust representation learning. Specifically, we first convert time series into the complex frequency domain using the Discrete Fourier Transform (DFT). The Transformer architecture is then applied to the frequency spectra to capture cross-variate dependencies, with the real and imaginary parts processed independently. However, we observe that the vanilla attention matrix exhibits a low-rank characteristic, thus limiting representation diversity. This could be attributed to the inherent sparsity of the frequency domain and the strong-value-focused nature of Softmax in vanilla attention. To address this, we enhance the vanilla attention mechanism by introducing an additional learnable matrix to the original attention matrix, followed by row-wise L1 normalization. Theoretical analysis~demonstrates that this enhanced attention mechanism improves both feature diversity and gradient flow. Extensive experiments demonstrate that FreEformer consistently outperforms state-of-the-art models on eighteen real-world benchmarks covering electricity, traffic, weather, healthcare and finance. Notably, the enhanced attention mechanism also consistently improves the performance of state-of-the-art Transformer-based forecasters.

FreEformer: Frequency Enhanced Transformer for Multivariate Time Series Forecasting

TL;DR

<3-5 sentence high-level summary> FreEformer tackles multivariate time series forecasting by leveraging a frequency-domain representation of the data, applying a Transformer to frequency-domain tokens to capture cross-variate dependencies. The key novelty is an enhanced attention mechanism that adds a learnable matrix to the attention scores and normalizes rows, increasing representation diversity and improving gradient flow. The approach is theoretically motivated and empirically validated across 18 real-world datasets, achieving state-of-the-art performance and demonstrating that the enhanced attention can also boost other Transformer forecasters as a plug-in. The work presents a simple yet powerful baseline that highlights the benefits of frequency-domain modeling for robust, cross-variable forecasting.</p>

Abstract

This paper presents \textbf{FreEformer}, a simple yet effective model that leverages a \textbf{Fre}quency \textbf{E}nhanced Trans\textbf{former} for multivariate time series forecasting. Our work is based on the assumption that the frequency spectrum provides a global perspective on the composition of series across various frequencies and is highly suitable for robust representation learning. Specifically, we first convert time series into the complex frequency domain using the Discrete Fourier Transform (DFT). The Transformer architecture is then applied to the frequency spectra to capture cross-variate dependencies, with the real and imaginary parts processed independently. However, we observe that the vanilla attention matrix exhibits a low-rank characteristic, thus limiting representation diversity. This could be attributed to the inherent sparsity of the frequency domain and the strong-value-focused nature of Softmax in vanilla attention. To address this, we enhance the vanilla attention mechanism by introducing an additional learnable matrix to the original attention matrix, followed by row-wise L1 normalization. Theoretical analysis~demonstrates that this enhanced attention mechanism improves both feature diversity and gradient flow. Extensive experiments demonstrate that FreEformer consistently outperforms state-of-the-art models on eighteen real-world benchmarks covering electricity, traffic, weather, healthcare and finance. Notably, the enhanced attention mechanism also consistently improves the performance of state-of-the-art Transformer-based forecasters.
Paper Structure (56 sections, 4 theorems, 56 equations, 14 figures, 30 tables)

This paper contains 56 sections, 4 theorems, 56 equations, 14 figures, 30 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Transformer-Based Forecasters
  4. Frequency-Domain Forecasters
  5. Transformer Variants
  6. Preliminaries
  7. Method
  8. Overall Architecture
  9. Frequency-Domain Operations
  10. Prediction Head
  11. Enhanced Attention
  12. Theoretical Analysis
  13. Feature Diversity
  14. Gradient Flow
  15. Combination of MLP and Vanilla Attention
  16. ...and 41 more sections

Key Result

Theorem 1

Given the time series $\mathbf{x}\in \mathbb{R}^N$ and its corresponding frequency spectrum $\mathcal{F}\in \mathbb{C}^N$. Let $\mathbf{W} \in \mathbb{C}^{N \times N}$ denote a weight matrix and $\mathbf{b} \in \mathbb{C}^N$ a bias vector. Under these definitions, the following DFT pair holds: where Here, $\circledast$ denotes the circular convolution, and $\odot$ represents the Hadamard (elemen

Figures (14)

  • Figure 1: Time series and their corresponding frequency spectra. The series are normalized before applying the DFT, and the amplitudes of the frequency spectra are plotted. (1) The frequency spectra often exhibit strong consistency across adjacent temporal spans within the same time series, forming the basis for frequency-based forecasting. (2) Strong correlations between the two variables in PEMS04 (rows 2 and 3) are observed, suggesting that exploring such multivariate relationships could lead to more robust representations. (3) The frequency spectrum usually exhibits sparsity, with a few dominant frequencies.
  • Figure 2: Overall structure of the FreEformer. We leverage the frequency spectrum to capture temporal patterns and employ an enhanced Transformer to model dependencies among multivariate spectra. The enhanced Transformer introduces a learnable matrix to the attention mechanism, which, as shown through theoretical analysis, addresses potential low-rank issues and improves gradient flow.
  • Figure 3: Attention matrices from state-of-the-art forecasters on the Weather dataset. The FreEformer with vanilla attention typically exhibits a low rank, likely due to the inherent sparsity of the frequency spectrum and the strong-value-focused nature of the Softmax function in vanilla attention.
  • Figure 4: Attention matrices from vanilla and enhanced attention. The left column shows the low-rank attention matrix from the vanilla attention (Weather: 3, ECL: 137), with most entries near zero. The right three columns show the original attention matrix ($\mathbf{A}$), the learned addition matrix ($\mathrm{Softplus} (\mathbf{B})$), and the final attention matrix ($\mathrm{Norm} \left ( \mathbf{A}+\mathrm{Softplus} (\mathbf{B}) \right )$). The final matrix exhibits more prominent values and higher ranks (Weather: 21, ECL: 321).
  • Figure 5: Illustration of the Jacobian matrices of $\mathbf{c}$ with respect to $\mathbf{a}$ and $\mathbf{b}$ for the Weather and COVID-19 datasets.
  • ...and 9 more figures

Theorems & Definitions (6)

  • Theorem 1: Frequency-domain linear projection and time-domain convolutions
  • Theorem 2
  • Theorem 3: Frequency-domain linear projection and time-domain convolutions
  • proof
  • Theorem 4: Rank and condition number of matrix sums
  • proof