Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the dimension of pullback attractors in recurrent neural networks

Muhammed Fadera

TL;DR

This work addresses why reservoir computers can accurately reconstruct chaotic attractors and compute invariants despite high-dimensional state spaces. It treats RCs as nonautonomous dynamical systems driven by input sequences and derives a rigorous bound: the box-counting dimension of the pullback attractor is bounded above by the box-counting dimension of the input-sequence space, with equality possible when the input space has open support. For inputs from an $N_{in}$-dimensional system, the bound gives $ ext{dim}_B( ext{pullback attractor}) \\le N_{in}$, revealing effective low-dimensional reservoir dynamics. The paper also proves that if the input space contains an open set, the pullback attractor can contain an open subset and thus attain the full reservoir dimension $N_r$, providing a theoretical explanation for successful attractor reconstruction and computation of Lyapunov exponents. Numerically, the results are illustrated using ESNs driven by Lorenz and Rössler systems and by periodic inputs, validating the upper bounds and highlighting regimes where ESP may fail or the attractor fills the reservoir space.

Abstract

Recurrent neural networks trained via the reservoir computing paradigm have demonstrated remarkable success in learning and reconstructing attractors from chaotic systems, often replicating quantities such as Lyapunov exponents and fractal dimensions. It has recently been conjectured that this is because the reservoir computer embeds the dynamics of the chaotic system in its state space before learning. This conjecture has been established for reservoir computers with linear activation functions and remains open for more general reservoir systems. In this work, we employ a non-autonomous dynamical systems approach to establish an upper bound for the box-counting dimension of the pullback attractor, a subset of the reservoir state space that is approximated during training and prediction phases. We prove that the box-counting dimension of the pullback attractor is bounded above by the box-counting dimension of the space of input sequences with respect to the product topology. In particular, for input sequences originating from an Nin-dimensional smooth dynamical system or their generic continuously differentiable observations, the box-counting dimension of the pullback attractor is bounded above by Nin. The results obtained here highlight the fact that, while a reservoir computer may possess a very high-dimensional state space, it exhibits effective low-dimensional dynamics. Our findings also partly explain why reservoir computers are successful in tasks such as attractor reconstruction and the computation of dynamic invariants like Lyapunov exponents and fractal dimensions.

On the dimension of pullback attractors in recurrent neural networks

TL;DR

This work addresses why reservoir computers can accurately reconstruct chaotic attractors and compute invariants despite high-dimensional state spaces. It treats RCs as nonautonomous dynamical systems driven by input sequences and derives a rigorous bound: the box-counting dimension of the pullback attractor is bounded above by the box-counting dimension of the input-sequence space, with equality possible when the input space has open support. For inputs from an -dimensional system, the bound gives , revealing effective low-dimensional reservoir dynamics. The paper also proves that if the input space contains an open set, the pullback attractor can contain an open subset and thus attain the full reservoir dimension , providing a theoretical explanation for successful attractor reconstruction and computation of Lyapunov exponents. Numerically, the results are illustrated using ESNs driven by Lorenz and Rössler systems and by periodic inputs, validating the upper bounds and highlighting regimes where ESP may fail or the attractor fills the reservoir space.

Abstract

Recurrent neural networks trained via the reservoir computing paradigm have demonstrated remarkable success in learning and reconstructing attractors from chaotic systems, often replicating quantities such as Lyapunov exponents and fractal dimensions. It has recently been conjectured that this is because the reservoir computer embeds the dynamics of the chaotic system in its state space before learning. This conjecture has been established for reservoir computers with linear activation functions and remains open for more general reservoir systems. In this work, we employ a non-autonomous dynamical systems approach to establish an upper bound for the box-counting dimension of the pullback attractor, a subset of the reservoir state space that is approximated during training and prediction phases. We prove that the box-counting dimension of the pullback attractor is bounded above by the box-counting dimension of the space of input sequences with respect to the product topology. In particular, for input sequences originating from an Nin-dimensional smooth dynamical system or their generic continuously differentiable observations, the box-counting dimension of the pullback attractor is bounded above by Nin. The results obtained here highlight the fact that, while a reservoir computer may possess a very high-dimensional state space, it exhibits effective low-dimensional dynamics. Our findings also partly explain why reservoir computers are successful in tasks such as attractor reconstruction and the computation of dynamic invariants like Lyapunov exponents and fractal dimensions.
Paper Structure (14 sections, 11 theorems, 87 equations, 1 figure, 1 table)

This paper contains 14 sections, 11 theorems, 87 equations, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. Notation
  4. The space of input sequences
  5. Dynamics of recurrent neural networks
  6. RNNs as nonautonomous dynamical system
  7. Entire solutions, the Echo State Property and Pullback attraction
  8. Pullback Attractor as the image of a continuous map
  9. Dimension of the pullback attractor
  10. Upper bound for finite dimensional input sequence spaces
  11. Lower bound and an infinite-dimensional input sequence space
  12. Conditions of Theorem \ref{['thm:fractal_dim_H']} for ESNs
  13. Numerical Experiments
  14. Discussions and conclusions

Key Result

Theorem 2.1

Suppose that the map $g:U \times X \to X$ is uniformly a $\mu$-contraction in the state variable. Let $H_0 \in C(\mathcal{U}, X)$ and $H_n$ be defined as Then there exists a unique continuous map $H \in C(\mathcal{U}, X)$ such that $\lim_{n \to \infty} H_n = H$ with respect to the uniform metric $\rho$ on $C(\mathcal{U}, X)$ for all initial choice $H_0 \in C(\mathcal{U}, X)$.

Figures (1)

  • Figure 1: Box-counting dimension of the set of states from ESNs with $N_r = 10, 20$ and $30$ hidden units driving with the dynamics of the Lorenz (left) and Rössler attractors (right), as well as their first components $u_1$. The orange dash line is the estimate box-counting dimension for the trajectories of each system that was used to drive the ESNs. The dash black vertical line is the rate of contraction $\rho$ is equal to the inverse of the absolute value of the most negative Lyapunov exponents of each system.

Theorems & Definitions (29)

  • Definition 2.1: Skew product flow, KloedenNonautonomousSystems
  • Definition 2.2: The cocycle property
  • Definition 2.3
  • Definition 2.4
  • Definition 2.5: Echo state property
  • Remark 1
  • Definition 2.6: Pullback Attractor
  • Theorem 2.1
  • proof
  • Theorem 2.2
  • ...and 19 more