Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Synthetic Time Series Forecasting with Transformer Architectures: Extensive Simulation Benchmarks

Ali Forootani, Mohammad Khosravi

TL;DR

This work benchmarks three Transformer-based time-series models—PatchTST, Informer, and Autoformer—across three architectural variants (Minimal, Standard, Full) using 1500 controlled experiments on synthetic signals to assess accuracy, robustness, and scalability under clean and noisy conditions. It shows PatchTST Standard as the most robust generalist, Autoformer variants excelling with trend-seasonal decomposition, and Informer limited by noise sensitivity for long-horizon tasks. The authors further introduce Deep Koopformer, a stability-constrained Transformer-Koopman framework that enforces spectral stability and energy contraction in a learned latent space, improving long-horizon forecasts for nonlinear/dynamical systems such as Van der Pol and Lorenz. This hybrid approach combines the flexibility of deep sequence models with principled dynamical-system constraints, offering interpretable and robust forecasting suitable for noisy real-world conditions. Overall, the paper provides a comprehensive, standardized comparison and a promising direction toward dynamics-aware, Koopman-augmented transformers for reliable time-series prediction.

Abstract

Time series forecasting plays a critical role in domains such as energy, finance, and healthcare, where accurate predictions inform decision-making under uncertainty. Although Transformer-based models have demonstrated success in sequential modeling, their adoption for time series remains limited by challenges such as noise sensitivity, long-range dependencies, and a lack of inductive bias for temporal structure. In this work, we present a unified and principled framework for benchmarking three prominent Transformer forecasting architectures-Autoformer, Informer, and Patchtst-each evaluated through three architectural variants: Minimal, Standard, and Full, representing increasing levels of complexity and modeling capacity. We conduct over 1500 controlled experiments on a suite of ten synthetic signals, spanning five patch lengths and five forecast horizons under both clean and noisy conditions. Our analysis reveals consistent patterns across model families. To advance this landscape further, we introduce the Koopman-enhanced Transformer framework, Deep Koopformer, which integrates operator-theoretic latent state modeling to improve stability and interpretability. We demonstrate its efficacy on nonlinear and chaotic dynamical systems. Our results highlight Koopman based Transformer as a promising hybrid approach for robust, interpretable, and theoretically grounded time series forecasting in noisy and complex real-world conditions.

Synthetic Time Series Forecasting with Transformer Architectures: Extensive Simulation Benchmarks

TL;DR

This work benchmarks three Transformer-based time-series models—PatchTST, Informer, and Autoformer—across three architectural variants (Minimal, Standard, Full) using 1500 controlled experiments on synthetic signals to assess accuracy, robustness, and scalability under clean and noisy conditions. It shows PatchTST Standard as the most robust generalist, Autoformer variants excelling with trend-seasonal decomposition, and Informer limited by noise sensitivity for long-horizon tasks. The authors further introduce Deep Koopformer, a stability-constrained Transformer-Koopman framework that enforces spectral stability and energy contraction in a learned latent space, improving long-horizon forecasts for nonlinear/dynamical systems such as Van der Pol and Lorenz. This hybrid approach combines the flexibility of deep sequence models with principled dynamical-system constraints, offering interpretable and robust forecasting suitable for noisy real-world conditions. Overall, the paper provides a comprehensive, standardized comparison and a promising direction toward dynamics-aware, Koopman-augmented transformers for reliable time-series prediction.

Abstract

Time series forecasting plays a critical role in domains such as energy, finance, and healthcare, where accurate predictions inform decision-making under uncertainty. Although Transformer-based models have demonstrated success in sequential modeling, their adoption for time series remains limited by challenges such as noise sensitivity, long-range dependencies, and a lack of inductive bias for temporal structure. In this work, we present a unified and principled framework for benchmarking three prominent Transformer forecasting architectures-Autoformer, Informer, and Patchtst-each evaluated through three architectural variants: Minimal, Standard, and Full, representing increasing levels of complexity and modeling capacity. We conduct over 1500 controlled experiments on a suite of ten synthetic signals, spanning five patch lengths and five forecast horizons under both clean and noisy conditions. Our analysis reveals consistent patterns across model families. To advance this landscape further, we introduce the Koopman-enhanced Transformer framework, Deep Koopformer, which integrates operator-theoretic latent state modeling to improve stability and interpretability. We demonstrate its efficacy on nonlinear and chaotic dynamical systems. Our results highlight Koopman based Transformer as a promising hybrid approach for robust, interpretable, and theoretically grounded time series forecasting in noisy and complex real-world conditions.

Paper Structure

This paper contains 82 sections, 1 theorem, 60 equations, 10 figures, 9 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries and Notation
  3. Time Series Data
  4. Forecasting Task
  5. Common Operations and Notation
  6. PatchTST Variants: Architecture and Computational Complexity
  7. PatchTST Minimal: A Lightweight Transformer Backbone
  8. Computational Complexity
  9. PatchTST Standard: Learning Data-Driven Positional Embeddings
  10. Computational Complexity
  11. PatchTST Full: A Flexible Encoder-Decoder Architecture
  12. Encoder
  13. Decoder
  14. Output
  15. Computational Complexity
  16. ...and 67 more sections

Key Result

Theorem 1

(Attention Complexity Reduction in ProbSparse Informer) If the attention mechanism is dominated by a small number of significant query-key pairs, ProbSparse reduces the attention complexity from $\mathcal{O}(L_Q L_K)$ to $\mathcal{O}(L_Q \log L_K)$ with bounded approximation error.

Figures (10)

  • Figure 1: 10 synthetic time series data used for benchmarking transformer-based forecasting models.
  • Figure 2: Performance of PatchTST variants (Minimal, Standard, Full) on various patch lengths and forecast horizons, averaged over all signals. Left column: RMSE heatmaps. Right column: MAE heatmaps.
  • Figure 3: Performance of Informer variants (Minimal, Standard, Full) on various patch lengths and forecast horizons, averaged over all signals.
  • Figure 4: Performance of Autoformer variants (Minimal, Standard, Full) on various patch lengths and forecast horizons, averaged over all signals.
  • Figure 5: Aggregated RMSE and MAE heatmaps compare Autoformer, Informer, and PatchTST across 10 synthetic signals, 5 patch lengths, and 5 forecast horizons. Autoformer leads overall, aided by trend-seasonal decomposition. PatchTST performs competitively, often matching or beating Autoformer in MAE. Informer shows higher errors, especially at longer horizons. Global color normalization ensures fair comparison.
  • ...and 5 more figures

Theorems & Definitions (3)

  • Remark 1
  • Theorem 1
  • proof