Learning dynamical systems with hit-and-run random feature maps

TL;DR

This work addresses forecasting chaotic dynamical systems using tanh-based random feature maps (RFMs) with fixed internal weights. It introduces a data-informed hit-and-run initialization, skip connections, deep stacking, and localization to overcome saturation, nonlinearity, and the curse of dimensionality, achieving state-of-the-art forecast skill with much smaller networks than reservoir computing approaches. The authors demonstrate strong single-trajectory forecasts and accurate long-time statistics on Lorenz-63, Lorenz-96, and Kuramoto-Sivashinsky, with depth and localization providing notable gains and scalable training times. The approach is computationally efficient, requires tuning only a single hyperparameter in many cases, and is complemented by open-source code and detailed appendices on algorithms and localization schemes. Together, these findings position deep/localized RFMs as competitive, scalable surrogates for data-driven forecasting of high-dimensional chaotic dynamics, with potential for integration with data assimilation and partial-noise scenarios.

Abstract

We show how random feature maps can be used to forecast dynamical systems with excellent forecasting skill. We consider the tanh activation function and judiciously choose the internal weights in a data-driven manner such that the resulting features explore the nonlinear, non-saturated regions of the activation function. We introduce skip connections and construct a deep variant of random feature maps by combining several units. To mitigate the curse of dimensionality, we introduce localization where we learn local maps, employing conditional independence. Our modified random feature maps provide excellent forecasting skill for both single trajectory forecasts as well as long-time estimates of statistical properties, for a range of chaotic dynamical systems with dimensions up to 512. In contrast to other methods such as reservoir computers which require extensive hyperparameter tuning, we effectively need to tune only a single hyperparameter, and are able to achieve state-of-the-art forecast skill with much smaller networks.

Figures (19)

• Figure 1: Illustration of the types of features produced by a $\tanh$-activation function, motivating the choice of the internal weights and biases $(\mathbf{W}_{\rm in},\mathbf{b_{\rm in}})$. Here and elsewhere $L_0=0.4$ and $L_1=3.5$.
• Figure 2: Schematic of the deep architecture DeepSkip with depth $B=3$. The symbols $\|$, $=$ and $+$ denote concatenation, identity operation and addition (skip connection), respectively.
• Figure 3: Schematic of a localized architecture. In this example, the local state dimension is $G=3$ and the interaction length is $I=2$.
• Figure 4: An example of a forecast by a DeepSkip model with width $D_r=1,024$ and depth $B=16$ for the L63 system \ref{['eq:L63']}. The surrogate model is able to forecast accurately up to ${\rm{VPT}}\approx 19$ Lyapunov time units.
• Figure 5: Kernel density plots of VPT for the L63 system \ref{['eq:L63']} for $(N, \Delta t, \varepsilon)=(5\times10^4, 0.01, 0.3)$. For RFM and SkipRFM, $\log_2(D_r)$ is indicated on the top of the plots. For DeepSkip, $(\log_2(D_r), B)$ is indicated on the top of each plot. The $*$-symbol indicates the model with the best mean VPT within each architecture.