Delay Embedding Theory of Neural Sequence Models

TL;DR

This work investigates why neural sequence models can infer unobserved dynamics by framing models as delay embeddings grounded in dynamical systems theory. By training 1-layer decoder-transformers and Linear Recurrent Unit (LRU) state-space models on next-step prediction for a noisy, partially observed Lorenz attractor, the authors quantify embedding quality using decoding accuracy, neighborhood smoothness, and unfolding metrics. They find that SSMs possess a stronger inductive bias for delay embeddings and yield better attractor reconstruction with lower dynamics-prediction error, though they can be more sensitive to observational noise; transformers can still learn viable embeddings with sufficient data. Overall, the study bridges delay embedding theory and deep learning sequence models, revealing architecture-dependent trade-offs that inform time-series forecasting and low-data regime designs, and highlighting parameter-efficiency advantages of SSMs in certain settings.

Abstract

To generate coherent responses, language models infer unobserved meaning from their input text sequence. One potential explanation for this capability arises from theories of delay embeddings in dynamical systems, which prove that unobserved variables can be recovered from the history of only a handful of observed variables. To test whether language models are effectively constructing delay embeddings, we measure the capacities of sequence models to reconstruct unobserved dynamics. We trained 1-layer transformer decoders and state-space sequence models on next-step prediction from noisy, partially-observed time series data. We found that each sequence layer can learn a viable embedding of the underlying system. However, state-space models have a stronger inductive bias than transformers-in particular, they more effectively reconstruct unobserved information at initialization, leading to more parameter-efficient models and lower error on dynamics tasks. Our work thus forges a novel connection between dynamical systems and deep learning sequence models via delay embedding theory.

Figures (9)

• Figure 1: Delay Embeddings. a. Noisy data from the $x$ dimension of the Lorenz attractor, on which our models are trained. b. Visualization of the top two Principal Components of a delay embedding with too few delays. Here, noise is amplified and the attractor is distorted. c. As in b, with too many delays. Here the attractor is folded into too many dimensions, making the data harder to model. d. In the intermediate, the embedding both reduces noise and has a geometry that reflects the original space (visualized in Appendix \ref{['lorenz']}).
• Figure 2: Learning curves, inset zoomed in. Shading indicates standard error.
• Figure 3: Sample top 4 Principal Components after training, of each sequence layer output, colored by trajectory.
• Figure 4: Delay embedding metrics across training, colored by architecture and dimension. a. MLP decoding of unobserved variables, test $R^2$. b. Linear decoding test $R^2$. c. Neighbors overlap fraction between full dynamic state and embedding. d. Conditional variance of future data given the embedding, averaged over future time steps from 1 to 10. Shading indicates standard error.
• Figure 5: 3-dimensional visualization of the Lorenz attractor. Simulated with noise, colored by time.