On the attractor in a high-dimensional neural network dynamics of reservoir computing: Lyapunov analysis viewpoint

TL;DR

The paper addresses how reservoir computing preserves the dynamical invariants of a target system by comparing the Lyapunov spectrum of a high-dimensional reservoir to that of the Hénon map, and by examining Lyapunov vectors restricted to the inertial manifold. It employs a setting where ${\mathbf A}=\rho{\mathbf A'}$ and analyzes how the spectral radius $\rho$ affects the embedding and stability, using both standard Lyapunov exponents and covariant Lyapunov vectors. The key contributions show that the leading Lyapunov exponent ${\Lambda}_1$ is faithfully reproduced for all $\rho$, while the second exponent ${\Lambda}_2$ can be recovered on the inertial manifold via $\tilde{\lambda}_2$ even when transversal directions are not strongly contracting; the study provides an algorithm to compute exponents within the inertial manifold and demonstrates the existence of a low-dimensional attractor embedded in the reservoir space. These results offer practical guidance for tuning $\rho$ to achieve faithful long-term dynamics in data-driven reservoir models and deepen the understanding of the geometric structure underlying reservoir computing.

Abstract

Recent theoretical developments of reservoir computing have clarified a sufficient condition about which reservoir computing can capture the dynamics of a target system, enabling the reconstruction of dynamical invariants. Even when the condition is relaxed, the reservoir computing is found to succeed in reconstructing time series. In this study, we investigate numerically the dynamical structures underlying the embedding structure by comparing the Lyapunov spectrum of a high-dimensional neural network in a reservoir computing model with that of the actual system. We also compute Lyapunov exponents restricted to the tangent space of the inertial manifold in a high-dimensional neural network. Our results provide numerical evidence that reservoir computing can accurately identify the Lyapunov spectrum of the target system, including all negative exponents.

Figures (9)

• Figure 1: Inference of a time-series and reconstruction of the attractor. In the top panel, a short-term trajectory of the $x$ variable for models with spectral radius $\rho=0.1$ (left) and $0.209$ (right) in red, alongside a trajectory of the $x$ variables of the actual Hénon map in blue. The middle panel shows the attractor of each constructed model. The bottom panel shows an enlarged view of the upper-left part of the corresponding upper panel.
• Figure 2: Box counting dimension of an attractor in the reservoir space with respect to the spectral radius $\rho$. The box counting dimension of an attractor in the reservoir space is estimated from a long-term time series. The black dashed line represents the attractor dimension (1.258) of the actual Hénon map. For the actual Hénon map, the Lyapunov and box-counting dimensions exhibit similar values. The dimension of the attractor in each model is less than two. When $\rho$ is larger than $0.1$, the box counting dimension deviates from line (1.258), indicating that a model with a large $\rho$ generates a slightly fat attractor.
• Figure 3: Lyapunov spectrum {$\lambda_i$} of the constructed data-driven model for the cases $\rho=0.001, 0.01, 0.1$. The first Lyapunov exponent $\Lambda_1$ is reconstructed for each case. For $\rho=0.001$ or $0.01$, the second Lyapunov exponent $\lambda_2(\rho)$ corresponds to the second $\Lambda_2$ of the actual Hénon map. For $\rho=0.1$, the third Lyapunov exponent $\lambda_3(\rho)$ corresponds to $\Lambda_2$. All Lyapunov exponents except the first one are negative.
• Figure 4: Model's Lyapunov exponents $\lambda_i$ as a function of the spectral radius $\rho$ of matrix $\mathbf{A}$. The model's first Lyapunov exponent in reservoir space $\lambda_{1}$, consistently reconstructs the actual first Lyapunov exponent $\Lambda_{1}$. Among the Lyapunov exponents in reservoir space $\lambda_{1}, \lambda_{2}, \lambda _{3}$, a Lyapunov exponent reconstructs the actual second Lyapunov exponent $\Lambda_{2}$ for models with varying spectral radius of matrix $\mathbf{A}$.
• Figure 5: Schematic of the inertial manifold. Original dynamics can be realized on the low-dimensional inertial manifold in the high-dimensional reservoir space. The angle between the Lyapunov vector and the tangent space of the inertial manifold, deviation angles were calculated.