Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Transformer Multivariate Forecasting: Less is More?

Jingjing Xu, Caesar Wu, Yuan-Fang Li, Pascal Bouvry

TL;DR

This paper proposes a novel transformer forecasting framework enhanced by Principal Component Analysis (PCA) to enhance transformer-based time series forecasting for intricate data and demonstrates the framework's ability to minimize prediction errors across all models and datasets while significantly reducing runtime.

Abstract

In the domain of multivariate forecasting, transformer models stand out as powerful apparatus, displaying exceptional capabilities in handling messy datasets from real-world contexts. However, the inherent complexity of these datasets, characterized by numerous variables and lengthy temporal sequences, poses challenges, including increased noise and extended model runtime. This paper focuses on reducing redundant information to elevate forecasting accuracy while optimizing runtime efficiency. We propose a novel transformer forecasting framework enhanced by Principal Component Analysis (PCA) to tackle this challenge. The framework is evaluated by five state-of-the-art (SOTA) models and four diverse real-world datasets. Our experimental results demonstrate the framework's ability to minimize prediction errors across all models and datasets while significantly reducing runtime. From the model perspective, one of the PCA-enhanced models: PCA+Crossformer, reduces mean square errors (MSE) by 33.3% and decreases runtime by 49.2% on average. From the dataset perspective, the framework delivers 14.3% MSE and 76.6% runtime reduction on Electricity datasets, as well as 4.8% MSE and 86.9% runtime reduction on Traffic datasets. This study aims to advance various SOTA models and enhance transformer-based time series forecasting for intricate data. Code is available at: https://github.com/jingjing-unilu/PCA_Transformer.

Transformer Multivariate Forecasting: Less is More?

TL;DR

This paper proposes a novel transformer forecasting framework enhanced by Principal Component Analysis (PCA) to enhance transformer-based time series forecasting for intricate data and demonstrates the framework's ability to minimize prediction errors across all models and datasets while significantly reducing runtime.

Abstract

In the domain of multivariate forecasting, transformer models stand out as powerful apparatus, displaying exceptional capabilities in handling messy datasets from real-world contexts. However, the inherent complexity of these datasets, characterized by numerous variables and lengthy temporal sequences, poses challenges, including increased noise and extended model runtime. This paper focuses on reducing redundant information to elevate forecasting accuracy while optimizing runtime efficiency. We propose a novel transformer forecasting framework enhanced by Principal Component Analysis (PCA) to tackle this challenge. The framework is evaluated by five state-of-the-art (SOTA) models and four diverse real-world datasets. Our experimental results demonstrate the framework's ability to minimize prediction errors across all models and datasets while significantly reducing runtime. From the model perspective, one of the PCA-enhanced models: PCA+Crossformer, reduces mean square errors (MSE) by 33.3% and decreases runtime by 49.2% on average. From the dataset perspective, the framework delivers 14.3% MSE and 76.6% runtime reduction on Electricity datasets, as well as 4.8% MSE and 86.9% runtime reduction on Traffic datasets. This study aims to advance various SOTA models and enhance transformer-based time series forecasting for intricate data. Code is available at: https://github.com/jingjing-unilu/PCA_Transformer.
Paper Structure (16 sections, 5 figures, 7 tables, 1 algorithm)

This paper contains 16 sections, 5 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Contributions
  3. Related Work
  4. Transformer in Time Series Forecasting
  5. PCA
  6. PCA+Transformer in Time Series Forecasting
  7. Experiments
  8. Dataset
  9. Experimental Formulation
  10. Experimental Environment and Configuration
  11. Framework Construction
  12. Experimental Results with Analysis
  13. Model Perspective
  14. Dataset Perspective
  15. Conclusion and Future Works
  16. ...and 1 more sections

Figures (5)

  • Figure 1: Trend of transformer in time series forecasting topic from 2017.
  • Figure 2: Structure overview of the PCA-enhanced transformer forecasting framework.
  • Figure 3: Simplified vanilla transformer architecture with three main components (Attention, Add&Norm, FeedForward).
  • Figure 4: Left: PCC of ETTh1 dataset. Right: PCC of Weather dataset.
  • Figure 5: Left: PCC of Electricity dataset. Right: PCC of Traffic dataset.