Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learnable Koopman-Enhanced Transformer-Based Time Series Forecasting with Spectral Control

Ali Forootani, Raffaele Iervolino

TL;DR

This work addresses long-horizon time-series forecasting by integrating learnable Koopman operators into Transformer-based forecasters, enabling explicit spectral control of latent dynamics. It introduces four operator families under an orthogonal–diagonal–orthogonal (ODO) factorization, combined with Lyapunov regularization, and demonstrates their integration with PatchTST, Autoformer, and Informer. The approach yields stable, interpretable, and invertible latent dynamics across diverse domains (climate, crypto, energy) with favorable bias–variance trade-offs and robust cross-configuration performance. Spectral diagnostics reveal that constrained and learnable variants maintain contractive yet expressive spectra within a mid-range band, sustaining reliable long-range forecasts. Overall, the Learnable-DeepKoopFormer offers a principled, scalable framework that unifies operator theory with modern forecasting architectures for principled, robust sequence modelling.

Abstract

This paper proposes a unified family of learnable Koopman operator parameterizations that integrate linear dynamical systems theory with modern deep learning forecasting architectures. We introduce four learnable Koopman variants-scalar-gated, per-mode gated, MLP-shaped spectral mapping, and low-rank Koopman operators which generalize and interpolate between strictly stable Koopman operators and unconstrained linear latent dynamics. Our formulation enables explicit control over the spectrum, stability, and rank of the linear transition operator while retaining compatibility with expressive nonlinear backbones such as Patchtst, Autoformer, and Informer. We evaluate the proposed operators in a large-scale benchmark that also includes LSTM, DLinear, and simple diagonal State-Space Models (SSMs), as well as lightweight transformer variants. Experiments across multiple horizons and patch lengths show that learnable Koopman models provide a favorable bias-variance trade-off, improved conditioning, and more interpretable latent dynamics. We provide a full spectral analysis, including eigenvalue trajectories, stability envelopes, and learned spectral distributions. Our results demonstrate that learnable Koopman operators are effective, stable, and theoretically principled components for deep forecasting.

Learnable Koopman-Enhanced Transformer-Based Time Series Forecasting with Spectral Control

TL;DR

This work addresses long-horizon time-series forecasting by integrating learnable Koopman operators into Transformer-based forecasters, enabling explicit spectral control of latent dynamics. It introduces four operator families under an orthogonal–diagonal–orthogonal (ODO) factorization, combined with Lyapunov regularization, and demonstrates their integration with PatchTST, Autoformer, and Informer. The approach yields stable, interpretable, and invertible latent dynamics across diverse domains (climate, crypto, energy) with favorable bias–variance trade-offs and robust cross-configuration performance. Spectral diagnostics reveal that constrained and learnable variants maintain contractive yet expressive spectra within a mid-range band, sustaining reliable long-range forecasts. Overall, the Learnable-DeepKoopFormer offers a principled, scalable framework that unifies operator theory with modern forecasting architectures for principled, robust sequence modelling.

Abstract

This paper proposes a unified family of learnable Koopman operator parameterizations that integrate linear dynamical systems theory with modern deep learning forecasting architectures. We introduce four learnable Koopman variants-scalar-gated, per-mode gated, MLP-shaped spectral mapping, and low-rank Koopman operators which generalize and interpolate between strictly stable Koopman operators and unconstrained linear latent dynamics. Our formulation enables explicit control over the spectrum, stability, and rank of the linear transition operator while retaining compatibility with expressive nonlinear backbones such as Patchtst, Autoformer, and Informer. We evaluate the proposed operators in a large-scale benchmark that also includes LSTM, DLinear, and simple diagonal State-Space Models (SSMs), as well as lightweight transformer variants. Experiments across multiple horizons and patch lengths show that learnable Koopman models provide a favorable bias-variance trade-off, improved conditioning, and more interpretable latent dynamics. We provide a full spectral analysis, including eigenvalue trajectories, stability envelopes, and learned spectral distributions. Our results demonstrate that learnable Koopman operators are effective, stable, and theoretically principled components for deep forecasting.
Paper Structure (37 sections, 8 theorems, 37 equations, 15 figures, 1 algorithm)

This paper contains 37 sections, 8 theorems, 37 equations, 15 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Koopman Operators and Deep Learning
  3. Contributions: Learnable Koopman--Transformer Framework
  4. Learnable-DeepKoopFormer Architecture
  5. Learnable Koopman Variants
  6. Theoretical Properties of the Learnable Koopman Propagator
  7. Spectral stability, contraction, and invertibility
  8. Expressiveness of spectrally bounded and low-rank operators
  9. Low-rank structure and approximation
  10. Lyapunov regularization and energy decay
  11. Theoretical stability advantage of the Koopman layer
  12. (i) Stability of the Koopman parameterisation
  13. (ii) Unstable but loss-equivalent SSM solutions
  14. Invertibility in finite dimensions and Banach space extension
  15. Numerical Simulations
  16. ...and 22 more sections

Key Result

Proposition 1

Let $\mathcal{K}_\phi$ be parameterized as in eq:koop_odo, with orthonormal $U_\phi, V_\phi$ and diagonal $\mathrm{diag}(\Sigma_\phi)$ satisfying eq:rho_max_bound. Then where $\rho(\cdot)$ denotes the spectral radius and $\|\cdot\|_2$ the spectral norm.

Figures (15)

  • Figure 1: Violin plots of train/test MSE and MAE for all architectures on the wind speed forecasting task, aggregated over all patch lengths and horizons.
  • Figure 2: Spectral distributions of latent Koopman operators for CIMIP6 wind speed forecasting task.
  • Figure 3: Violin plots of train/test MSE and MAE for all architectures on the pressure surface forecasting task, aggregated over all patch lengths and horizons.
  • Figure 4: Spectral distributions of latent Koopman operators for CIMIP6 pressure surface forecasting task.
  • Figure 5: Violin plots of train/test MSE and MAE for all architectures on the Cryptocurrency forecasting task, aggregated over all patch lengths and horizons.
  • ...and 10 more figures

Theorems & Definitions (16)

  • Proposition 1: Spectral stability
  • proof
  • Corollary 1: Exponential contraction in latent space
  • proof
  • Proposition 2: Invertibility and stability of the inverse
  • proof
  • Proposition 3: Representation of all spectrally bounded operators
  • proof
  • Proposition 4: Low-rank structure and norm bound
  • proof
  • ...and 6 more