Generative learning for nonlinear dynamics

TL;DR

The work addresses how classical nonlinear dynamics concepts—information production, attractor reconstruction, and symbolic dynamics—inform contemporary generative learning for complex time series. It integrates Takens-style embedding, Koopman lifting, and entropy-based perspectives within modern latent-variable and diffusion-style frameworks to highlight how latent representations can capture, propagate, and distill chaotic dynamics. Key contributions include reframing attractor reconstruction as density estimation in latent space, outlining lifting strategies to linearize nonlinear dynamics, and connecting entropy production with interpretability through discrete latent generators and nonequilibrium thermodynamics. The significance lies in providing inductive biases, theoretical links, and practical guidance for designing scalable, interpretable generative models that faithfully represent dynamical structures across diverse complex systems.

Abstract

Modern generative machine learning models demonstrate surprising ability to create realistic outputs far beyond their training data, such as photorealistic artwork, accurate protein structures, or conversational text. These successes suggest that generative models learn to effectively parametrize and sample arbitrarily complex distributions. Beginning half a century ago, foundational works in nonlinear dynamics used tools from information theory to infer properties of chaotic attractors from time series, motivating the development of algorithms for parametrizing chaos in real datasets. In this perspective, we aim to connect these classical works to emerging themes in large-scale generative statistical learning. We first consider classical attractor reconstruction, which mirrors constraints on latent representations learned by state space models of time series. We next revisit early efforts to use symbolic approximations to compare minimal discrete generators underlying complex processes, a problem relevant to modern efforts to distill and interpret black-box statistical models. Emerging interdisciplinary works bridge nonlinear dynamics and learning theory, such as operator-theoretic methods for complex fluid flows, or detection of broken detailed balance in biological datasets. We anticipate that future machine learning techniques may revisit other classical concepts from nonlinear dynamics, such as transinformation decay and complexity-entropy tradeoffs.

Figures (4)

• Figure 1: (Left) An example of a complex probability distribution $p(\mathbf{x})$ and a schematic of a simplified Markov Chain Monte Carlo sampling scheme. (Right) The natural measure $\mu(\mathbf{x})$ of a strange attractor, and a schematic of the divergence of a set of initial conditions. In these examples, $p(\mathbf{x})$ is taken from the distribution of proteins learned by a variational autoencoder trained on amino acid sequences ding2019deciphering, while $\mu(\mathbf{x})$ comes from a reduced order deterministic model of a convective cell.
• Figure 2: Latent dynamics revisit classical attractor reconstruction. (A) Time-delay embeddings of a univariate time series representing the radial velocity of a flow, at three different Reynolds numbers ($R$) leading to turbulence. Poincare sections are shown below each embedding. (B) The latent space of an autoencoder neural network trained on weak turbulence ($R = 40$). The latent states are further embedded in two dimensions using t-distributed stochastic neighbor embedding (t-SNE). Shading indicates power dissipation, and connected states indicate equivalent flow configurations under a discrete symmetry operation. Panel A is reprinted from Brandstäter et al. 1983. Panel B is reprinted from Page, Brenner, and Kerswell, 2021.brandstater1983lowpage2021revealing
• Figure 3: State space models generate complex dynamics. (A) Components of a generic state space model. (B) In a variant of Sparse Identification of Nonlinear Dynamics champion2019data, multilayer perceptrons deterministically transform high-dimensional observations to a low-dimensional latent space, in which the dynamics are propagated using analytical differential equations learned via sparse regression from a library of known functions. (C) In Latent Factor Analysis via Dynamical Systems pandarinath2018inferring, neuron spiking time series are deterministically encoded into latent initial conditions, which are evolved using a second recurrent neural network, and then decoded into latent factor time series. These latent factors parameterize the stochastic firing rate of an inhomogenous Poisson process. (D) In Manifold Interpolating Optimal-Transport Flows huguet2022manifold, high-dimensional gene expression measurements are encoded to a latent distribution that preserves the manifold diffusion distance. The latent measure is then propagated with optimal transport.
• Figure 4: Latent discretization and interpretability. (A) Successive stages of an adaptive approximation algorithm that fits locally-linear dynamics to parts of the phase space of a chaotic system. (B) A continuous-valued learning model that creates a discrete, latent self-organizing map from a continuous time series of sleep recordings. (C) The topological complexity of probabilistic automata fitted to a dynamical map across a range of chaotic and periodic regimes, plotted against the entropy of the time series. The most structurally-complex automaton occurs when the dynamics exhibit intermediate entropy. Panel A is reprinted from Costa, Ahamed, Stephens 2019. Panel B is reprinted from Huijben et al. 2023. Panel C is reprinted from Young & Crutchfield, 1989.costa2019adaptivehuijben2023somcrutchfield1989inferring.