Zero-shot forecasting of chaotic systems

TL;DR

Problem: Can zero-shot forecasting with large, domain-agnostic time-series models generalize to chaotic dynamical systems without task-specific training? Approach: Benchmark Chronos on $135$ chaotic systems, using short-term forecast metrics like $VPT$ and $sMAPE$, and long-term attractor metrics such as fractal dimension $d_{frac}$ and KL divergence $D_{stsp}$, across varying context lengths and with shuffled-context tests to probe in-context learning. Findings: larger Chronos models achieve short-term forecasts competitive with fully-trained baselines, excel at long-term attractor reconstruction, and exhibit in-context learning via context parroting, even when context order is scrambled; performance degrades under nonstationarity. Significance: demonstrates a scalable, training-free forecasting paradigm for complex nonlinear dynamics with potential implications for SciML and nonlinear dynamics, and points to future work on robustness and task-tuned adaptations.

Abstract

Time-series forecasting is a challenging problem that traditionally requires specialized models custom-trained for the specific task at hand. Recently, inspired by the success of large language models, foundation models pre-trained on vast amounts of time-series data from diverse domains have emerged as a promising candidate for general-purpose time-series forecasting. The defining characteristic of these foundation models is their ability to perform zero-shot learning, that is, forecasting a new system from limited context data without explicit re-training or fine-tuning. Here, we evaluate whether the zero-shot learning paradigm extends to the challenging task of forecasting chaotic systems. Across 135 distinct chaotic dynamical systems and $10^8$ timepoints, we find that foundation models produce competitive forecasts compared to custom-trained models (including NBEATS, TiDE, etc.), particularly when training data is limited. Interestingly, even after point forecasts fail, large foundation models are able to preserve the geometric and statistical properties of the chaotic attractors. We attribute this success to foundation models' ability to perform in-context learning and identify context parroting as a simple mechanism used by these models to capture the long-term behavior of chaotic dynamical systems. Our results highlight the potential of foundation models as a tool for probing nonlinear and complex systems.

Figures (16)

• Figure 1: Chaos as a benchmark for zero-shot forecasting of time series. We use $135$ distinct chaotic systems to generate chaotic trajectories from $20$ different initial conditions each. Each trajectory is used to train the baseline deep-learning models (NBEATS, TiDE, etc.) and also provided as context to the pre-trained LLM (we use Chronos, a best-in-class foundation model for time series). Both the trained baseline models and Chronos are then asked to predict the trajectory into the future. We measure the quality of the predictions in terms of both short-term accuracy and long-term attractor reconstruction. Across $10^4$ distinct trajectories and $10^8$ data points, we find that zero-shot forecasts can be competitive in both short-term predictions and in capturing the long-term "climate" of the dynamics.
• Figure 2: Zero-shot forecasts of chaotic systems. We use Chronos to predict the $x(t)$ and $y(t)$ components of the Lorenz oscillator. The zero-shot forecasts match remarkably well with the ground truth for both short-term prediction and long-term attractor reconstruction.
• Figure 3: Zero-shot models of chaotic systems are competitive with custom-trained models. Zero-shot forecasts from Chronos for five different model sizes (left), compared to other forecast models directly trained on the points given to Chronos as context (right). Inset plots show the valid prediction times (VPT), the first time each forecast exceeds an error limit. All error bars are over $135$ chaotic systems, each with $20$ distinct initial conditions.
• Figure 4: Zero-shot forecast models effectively capture attractor geometry. (A) Example forecasts produced by the zero-shot and trained models, for $20$ initial conditions from the Lorenz chaotic attractor. (B) The correlation between the fractal dimension of the predicted attractor and the true attractor (Spearman's rank-order coefficient, $N = 2420$ points, $p < 10^{-3}$ for all cases), versus the VPT of the corresponding model. The red markers represent variants of Chronos with different model sizes: tiny ($8M$ parameters), mini ($20M$), small ($46M$), base ($200M$), and large ($710M$). The blue markers represent the baseline models. Models closer to the top capture the attractor geometry better and models closer to the right make accurate point forecasts for longer. Error bars are standard errors over $135$ dynamical systems, each with $20$ different initial conditions.
• Figure 5: Context parroting as a mechanism for zero-shot forecasting. (A) Better zero-shot forecasts often have initial stages that overlap with the context. The context overlap quantifies the similarity between the last $30$ points of the context and the prior points. (B) Comparison of context overlap of the zero-shot forecasts (Chronos-base) with the best performing fully-trained model (NBEATS). The zero-shot model correlates with context significantly more than the trained models across the chaotic systems dataset (matched t-test, $N=135$, $p < 10^{-3}$).