Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting

Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, Rong Jin

TL;DR

Long-term time-series forecasting with Transformers faces high computational cost and limited global pattern capture. FEDformer integrates seasonal-trend decomposition with frequency-domain blocks (Fourier and Wavelet) and a mixture-of-experts decomposition to separate seasonal and trend components, enabling linear complexity. It achieves state-of-the-art accuracy across six benchmarks, with substantial reductions in MSE for both multivariate and univariate settings. The work provides theoretical justification for random Fourier component selection and extensive empirical analyses, including KS-distribution tests, to validate the design.

Abstract

Although Transformer-based methods have significantly improved state-of-the-art results for long-term series forecasting, they are not only computationally expensive but more importantly, are unable to capture the global view of time series (e.g. overall trend). To address these problems, we propose to combine Transformer with the seasonal-trend decomposition method, in which the decomposition method captures the global profile of time series while Transformers capture more detailed structures. To further enhance the performance of Transformer for long-term prediction, we exploit the fact that most time series tend to have a sparse representation in well-known basis such as Fourier transform, and develop a frequency enhanced Transformer. Besides being more effective, the proposed method, termed as Frequency Enhanced Decomposed Transformer ({\bf FEDformer}), is more efficient than standard Transformer with a linear complexity to the sequence length. Our empirical studies with six benchmark datasets show that compared with state-of-the-art methods, FEDformer can reduce prediction error by $14.8\%$ and $22.6\%$ for multivariate and univariate time series, respectively. Code is publicly available at https://github.com/MAZiqing/FEDformer.

FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting

TL;DR

Long-term time-series forecasting with Transformers faces high computational cost and limited global pattern capture. FEDformer integrates seasonal-trend decomposition with frequency-domain blocks (Fourier and Wavelet) and a mixture-of-experts decomposition to separate seasonal and trend components, enabling linear complexity. It achieves state-of-the-art accuracy across six benchmarks, with substantial reductions in MSE for both multivariate and univariate settings. The work provides theoretical justification for random Fourier component selection and extensive empirical analyses, including KS-distribution tests, to validate the design.

Abstract

Although Transformer-based methods have significantly improved state-of-the-art results for long-term series forecasting, they are not only computationally expensive but more importantly, are unable to capture the global view of time series (e.g. overall trend). To address these problems, we propose to combine Transformer with the seasonal-trend decomposition method, in which the decomposition method captures the global profile of time series while Transformers capture more detailed structures. To further enhance the performance of Transformer for long-term prediction, we exploit the fact that most time series tend to have a sparse representation in well-known basis such as Fourier transform, and develop a frequency enhanced Transformer. Besides being more effective, the proposed method, termed as Frequency Enhanced Decomposed Transformer ({\bf FEDformer}), is more efficient than standard Transformer with a linear complexity to the sequence length. Our empirical studies with six benchmark datasets show that compared with state-of-the-art methods, FEDformer can reduce prediction error by and for multivariate and univariate time series, respectively. Code is publicly available at https://github.com/MAZiqing/FEDformer.
Paper Structure (52 sections, 5 theorems, 26 equations, 8 figures, 13 tables)

This paper contains 52 sections, 5 theorems, 26 equations, 8 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Compact Representation of Time Series in Frequency Domain
  3. Model Structure
  4. FEDformer Framework
  5. Preliminary
  6. FEDformer Structure
  7. Fourier Enhanced Structure
  8. Discrete Fourier Transform (DFT)
  9. Frequency Enhanced Block with Fourier Transform (FEB-f)
  10. Frequency Enhanced Attention with Fourier Transform (FEA-f)
  11. Wavelet Enhanced Structure
  12. Discrete Wavelet Transform (DWT)
  13. Frequency Enhanced Block with Wavelet Transform (FEB-w)
  14. Frequency Enhanced Attention with Wavelet Transform (FEA-w)
  15. Mixture of Experts for Seasonal-Trend Decomposition
  16. ...and 37 more sections

Key Result

Theorem 1

Assume that $\mu(A)$, the coherence measure of matrix $A$, is $\Omega(k/n)$. Then, with a high probability, we have if $s = O(k^2/\epsilon^2)$.

Figures (8)

  • Figure 1: Different distribution between ground truth and forecasting output from vanilla Transformer in a real-world ETTm1 dataset. Left: frequency mode and trend shift. Right: trend shift.
  • Figure 2: FEDformer Structure. The FEDformer consists of $N$ encoders and $M$ decoders. The Frequency Enhanced Block (FEB, green blocks) and Frequency Enhanced Attention (FEA, red blocks) are used to perform representation learning in frequency domain. Either FEB or FEA has two subversions (FEB-f & FEB-w or FEA-f & FEA-w), where '-f' means using Fourier basis and '-w' means using Wavelet basis. The Mixture Of Expert Decomposition Blocks (MOEDecomp, yellow blocks) are used to extract seasonal-trend patterns from the input data.
  • Figure 3: Frequency Enhanced Block with Fourier transform (FEB-f) structure.
  • Figure 4: Frequency Enhanced Attention with Fourier transform (FEA-f) structure, $\sigma(\cdot)$ is the activation function.
  • Figure 5: Top Left: Wavelet frequency enhanced block decomposition stage. Top Right: Wavelet block reconstruction stage shared by FEB-w and FEA-w. Bottom: Wavelet frequency enhanced cross attention decomposition stage.
  • ...and 3 more figures

Theorems & Definitions (7)

  • Theorem 1
  • Definition 2.1
  • Theorem 2
  • Theorem 3
  • Theorem 4
  • Theorem 5
  • proof