Forecasting causal dynamics with universal reservoirs

TL;DR

This work develops a rigorous framework for forecasting causal dynamics with infinite memory using universal recurrent reservoirs, deriving explicit finite-horizon error bounds that scale as $e^{t(\lambda_1^{pos}+\delta)}$, where $\lambda_1^{pos}$ is the top Lyapunov exponent of the learned state-space system. It shows that, given an $\varepsilon$-accurate uniform approximation of the data-generating functional by a state-space learner, the forecast error in both state and observed variables can be tightly controlled for a finite horizon via a bound that incorporates Lipschitz constants and ergodic-transport constants, while accounting for Taylor expansion errors. The authors extend the analysis to the causal embedding forecasting scheme and illustrate the theory with numerical experiments on Lorenz and Rössler systems using echo state networks, demonstrating that the observed error growth aligns with the predicted Lyapunov-based rate. The results provide a bridge between universal approximation in input/output systems and rigorous forecasting performance for dynamical systems, offering practical bounds and a methodology for assessing forecast reliability in CCIMs and related infinite-memory settings.

Abstract

An iterated multistep forecasting scheme based on recurrent neural networks (RNN) is proposed for the time series generated by causal chains with infinite memory. This forecasting strategy contains, as a particular case, the iterative prediction strategies for dynamical systems that are customary in reservoir computing. Explicit error bounds are obtained as a function of the forecasting horizon, functional and dynamical features of the specific RNN used, and the approximation error committed by it. In particular, the growth rate of the error is shown to be exponential and controlled by the top Lyapunov exponent of the proxy system. The framework in the paper circumvents difficult-to-verify embedding hypotheses that appear in previous references in the literature and applies to new situations like the finite-dimensional observations of functional differential equations or the deterministic parts of stochastic processes to which standard embedding techniques do not necessarily apply.

Figures (5)

• Figure 1: Data from a trajectory used for training and forecasting the Lorenz system with access to full coordinatees. The first black line divides the washout from the trajectory data used in training the ESN. The second black line divides the training trajectory from the trajectory used to test forecasting.
• Figure 2: Prediction error for the Lorenz system with full access to system states, averaged over 100 trajectories. The error bound plotted has been averaged over the 100 trajectories. The gray band is an emperically calculated $95\%$ confidence interval for the errors committed by the $100$ trajectories. These are all plotted in the log scale.
• Figure 3: Predictions and errors for a single trajectory of an ESN trained on the first coordinate of the Lorenz system.
• Figure 4: Prediction error for the Lorenz system with access to the first coordinate, averaged over 100 trajectories. The error bound plotted has been averaged over the 100 trajectories. The gray band is an emperically calculated $95\%$ confidence interval for the errors committed by the $100$ trajectories. These are all plotted in the log scale.
• Figure 5: Prediction error for the Rössler system with full access to system states, averaged over 100 trajectories. The error bound plotted has been averaged over the 100 trajectories. The gray band is an emperically calculated $95\%$ confidence interval for the errors committed by the $100$ trajectories. These are all plotted in the log scale.

Theorems & Definitions (26)

• Proposition 2.1
• Corollary 2.2
• Theorem 2.3: Oseledets' Multiplicative Ergodic Theorem
• Definition 2.4
• Theorem 2.5
• Theorem 3.1
• Remark 3.2
• Remark 3.3
• Remark 3.4
• Remark 3.5