Transformer Learning of Chaotic Collective Dynamics in Many-Body Systems

TL;DR

This work shows that a self-attention-based transformer framework provides an effective approach for modeling such chaotic collective dynamics directly from time-series data, by selectively reweighting long-range temporal correlations and learning a non-Markovian reduced description that overcomes intrinsic limitations of conventional recurrent architectures.

Abstract

Learning reduced descriptions of chaotic many-body dynamics is fundamentally challenging: although microscopic equations are Markovian, collective observables exhibit strong memory and exponential sensitivity to initial conditions and prediction errors. We show that a self-attention-based transformer framework provides an effective approach for modeling such chaotic collective dynamics directly from time-series data. By selectively reweighting long-range temporal correlations, the transformer learns a non-Markovian reduced description that overcomes intrinsic limitations of conventional recurrent architectures. As a concrete demonstration, we study the one-dimensional semiclassical Holstein model, where interaction quenches induce strongly nonlinear and chaotic dynamics of the charge-density-wave order parameter. While pointwise predictions inevitably diverge at long times, the transformer faithfully reproduces the statistical "climate" of the chaos, including temporal correlations and characteristic decay scales. Our results establish self-attention as a powerful mechanism for learning effective reduced dynamics in chaotic many-body systems.

Figures (5)

• Figure 1: (a) A length-$T$ history of the order parameter $\Delta(t)$ is embedded, augmented with positional encoding, and processed by stacked transformer blocks with multi-head self-attention to predict the next value $\Delta(t+\delta t)$. (b) Multi-head self-attention computes relevance weights between different times via learned query--key interactions, enabling global temporal correlations across the input window.
• Figure 2: Two representative interaction-quench trajectories of the semiclassical Holstein model. The coordinate $x = i a$ is measured in lattice constant $a$, while time $t$ is measured in unit of time-step, which is set to $\delta t = 0.01 \Omega^{-1}$. Left: spatiotemporal evolution of the lattice displacement field $\{Q_i(t)\}$, showing mobile kink (domain-wall) defects separating CDW domains. Right: time evolution of the global CDW order parameter $\Delta_{\rho}(t)$, exhibiting irregular fluctuations driven by the collective kink dynamics.
• Figure 3: Prediction benchmark of the transformer model. (a) Scatter plot of the one-step prediction $\Delta_{T+1}^{\rm ML}$ versus the ground-truth value $\Delta_{T+1}^{\rm exact}$, showing excellent agreement along the diagonal. (b) Probability distribution of the prediction error $\delta=\Delta_{T+1}^{\rm ML}-\Delta_{T+1}^{\rm exact}$, which is sharply peaked around zero, indicating high predictive accuracy.
• Figure 4: Four representative trajectories of the global CDW order parameter $\Delta_\rho(t)$ from exact simulations (solid blue) and transformer predictions (dashed red). The model shows excellent short-time agreement, with long-time divergence due to error accumulation in the chaotic regime, while correctly capturing the statistical “climate” of the dynamics.
• Figure 5: Autocorrelation function of the global CDW order parameter $\Delta_\rho(t)$ computed from exact Ehrenfest dynamics (solid blue) and from transformer-generated trajectories (dashed red).