True Zero-Shot Inference of Dynamical Systems Preserving Long-Term Statistics

TL;DR

This work introduces DynaMix, the first zero-shot dynamical systems reconstruction (DSR) foundation model, achieving faithful replication of attractor geometry and long-term statistics from a short context without retraining. It leverages a multivariate mixture-of-experts built on Almost-Linear RNNs (AL-RNNs) pretrained across diverse DS, with a context-driven gating mechanism and a flexible context length. Through sparse teacher forcing and a compact training corpus, DynaMix attains superior zero-shot DSR performance and faster inference (≈$10^4$ parameters) compared to large TS foundation models, while delivering competitive or better short-term forecasts on real-world time series. The results warrant viewing DS-based principles as a powerful avenue to improve zero-shot forecasting, suggesting broader applicability of DS-informed foundation models for complex, real-world dynamics characterized by attractors and invariant statistics, even when empirical data are outside the training distribution. Future work points to incorporating nonstationary and multiscale dynamics, explicit filtering/decomposition modules, and irregular sampling to widen the applicability of DS-grounded foundations in TS forecasting.

Abstract

Complex, temporally evolving phenomena, from climate to brain activity, are governed by dynamical systems (DS). DS reconstruction (DSR) seeks to infer generative surrogate models of these from observed data, reproducing their long-term behavior. Existing DSR approaches require purpose-training for any new system observed, lacking the zero-shot and in-context inference capabilities known from LLMs. Here we introduce DynaMix, a novel multivariate ALRNN-based mixture-of-experts architecture pre-trained for DSR, the first DSR model able to generalize zero-shot to out-of-domain DS. Just from a provided context signal, without any re-training, DynaMix faithfully forecasts the long-term evolution of novel DS where existing time series (TS) foundation models, like Chronos, fail -- at a fraction of the number of parameters (0.1%) and orders of magnitude faster inference times. DynaMix outperforms TS foundation models in terms of long-term statistics, and often also short-term forecasts, even on real-world time series, like traffic or weather data, typically used for training and evaluating TS models, but not at all part of DynaMix' training corpus. We illustrate some of the failure modes of TS models for DSR problems, and conclude that models built on DS principles may bear a huge potential also for advancing the TS prediction field.

Figures (34)

• Figure 1: DynaMix achieves zero-shot DSR of attractor geometry and long-term temporal properties (power spectrum) from a short context signal while Chronos ansari2024chronos fails.
• Figure 2: Illustration of the DynaMix architecture. At test time, only a context signal $\bm{C}$ is provided and guides the selection of experts to yield arbitrarily long forward predictions of the dynamics.
• Figure 3: a) DynaMix zero-shot DSR (red) compared to ground truth (lightgray) using a $500$-step context (darkgray) for the Sprott M system. b) Zero-shot forecasts for the Selkov DS (true vector field in lightgray) from different initial conditions (red) outside the context range (darkgray). c) DSR quality as a function of context length for Lorenz-63. d) DSR quality as a function of the temporal resolution $\Delta t$ of the context signal. Error bands = STD
• Figure 4: Zero-shot DSR performance across all 54 test set DS for DynaMix and various TS foundation models for context length $T_C=2000$ (see Fig. \ref{['fig:performance_comparison_CL512']} for results with $T_C=512$). Median$\pm$MAD of $D_{stsp}$ (left, geometrical disagreement), $D_H$ (middle, temporal disagreement), and MASE (right, short-term prediction error).
• Figure 5: Number of model parameters vs. inference times for zero-shot generation of Lorenz-63 DS with $10000$ forecasting steps. All models were provided the exact same context and run on the same hardware. Note the log-scale on the $x$/$y$-axis. For comparison, also the time needed for training and inference of a custom-trained AL-RNN is shown (turquoise).