A Mechanistic Analysis of Transformers for Dynamical Systems

TL;DR

This work investigates the representational capabilities and limitations of single-layer Transformers when applied to dynamical data, and interprets causal self-attention as a linear, history-dependent recurrence and analyzes how it processes temporal information.

Abstract

Transformers are increasingly adopted for modeling and forecasting time-series, yet their internal mechanisms remain poorly understood from a dynamical systems perspective. In contrast to classical autoregressive and state-space models, which benefit from well-established theoretical foundations, Transformer architectures are typically treated as black boxes. This gap becomes particularly relevant as attention-based models are considered for general-purpose or zero-shot forecasting across diverse dynamical regimes. In this work, we do not propose a new forecasting model, but instead investigate the representational capabilities and limitations of single-layer Transformers when applied to dynamical data. Building on a dynamical systems perspective we interpret causal self-attention as a linear, history-dependent recurrence and analyze how it processes temporal information. Through a series of linear and nonlinear case studies, we identify distinct operational regimes. For linear systems, we show that the convexity constraint imposed by softmax attention fundamentally restricts the class of dynamics that can be represented, leading to oversmoothing in oscillatory settings. For nonlinear systems under partial observability, attention instead acts as an adaptive delay-embedding mechanism, enabling effective state reconstruction when sufficient temporal context and latent dimensionality are available. These results help bridge empirical observations with classical dynamical systems theory, providing insight into when and why Transformers succeed or fail as models of dynamical systems.

Figures (21)

• Figure 1: Schematic of the single-layer single-head self-attention Transformer architecture with optional linear output studied in this work. Adapted from raschka2023understanding.
• Figure 2: Transformer performance on linear SDOF systems. (a) Successful reproduction of oscillatory dynamics for $k=2000~\mathrm{N/m}$ with same-sign AR coefficients: the transformer accurately captures amplitude and phase, and preserves both the dominant modal peak in the frequency domain and the coherent modal decay ridge in the time--frequency representation. (b) Failure case for $k=500~\mathrm{N/m}$ with mixed-sign AR coefficients: the attention-only model produces over-smoothed responses and fails to recover the resonance peak and coherent modal ridge.
• Figure 3: Spectral analysis of transformer-predicted responses for the two-DOF linear system under different observability and temporal context settings. (a) Full observation with input sequence length 4: both modal frequencies ($f_1 = 4.1$ Hz, $f_2 = 9.5$ Hz) are accurately recovered. (b) Partial observation with input sequence length 4: distorted spectral content with failure to recover the true modal frequencies. (c) Partial observation with input sequence length 8: extending the temporal context enables recovery of both modal frequencies, indicating that temporal aggregation can partially compensate for limited spatial observability.
• Figure 4: (a) Phase portrait of the Van der Pol oscillator with $( \mu = 0.5 )$. The training (black), validation (green), and test (red) trajectories are shown. (b) Prediction error (|MSE|) under full observation across different models. The transformer performs comparably to the MLP, indicating that attention mechanisms do not confer a significant advantage when all state variables are observed, (c) Prediction error (|MSE|) under partial observation (only $x$ observable). The transformer significantly outperforms the MLP, demonstrating its ability to learn latent dynamics from delayed inputs.
• Figure 5: (a) Predicted time series on the limit cycle. Ground truth values (black circles) are compared against predictions from two transformer models with identical architectures: one without positional encoding (No P.E., red crosses) and one with learned positional encoding (P.E., cyan stars). (b) Latent correction term $Z$ plotted against $y_1(t)$, showing phase-dependent separation. (c) Visualization of $Z + y_1(t)$ against $y_1(t)$, the effective transformed input signal. (d) Latent space trajectories of 2D transformer models with and without positional encoding. Attention pattern visualizations across 10 trained models: (e) 1D MLP + Attention with learned P.E., (f) 2D MLP + Attention with learned P.E., and (g) 2D MLP + Attention without P.E.