Generative System Dynamics in Recurrent Neural Networks

TL;DR

This work analyzes continuous-time RNN dynamics, showing that skew-symmetric weight matrices $A$ combined with odd, bounded activations (notably $\tanh$) sustain perpetual oscillations without converging to fixed points. By constructing energy-like invariants and applying a Lyapunov-style argument, it demonstrates that limit cycles persist in nonlinear RNNs and can be characterized in both 2D and higher dimensions via invariant functions. Numerical simulations corroborate the theory, revealing that nonlinear activations can enhance numerical stability of forward Euler integration and that non-odd activations disrupt closed orbits, guiding architectural choices for robust temporal modeling. Overall, the results provide principled design insights for building RNNs capable of long-term memory and stable, oscillatory dynamics in continuous-time regimes.

Abstract

In this study, we investigate the continuous time dynamics of Recurrent Neural Networks (RNNs), focusing on systems with nonlinear activation functions. The objective of this work is to identify conditions under which RNNs exhibit perpetual oscillatory behavior, without converging to static fixed points. We establish that skew-symmetric weight matrices are fundamental to enable stable limit cycles in both linear and nonlinear configurations. We further demonstrate that hyperbolic tangent-like activation functions (odd, bounded, and continuous) preserve these oscillatory dynamics by ensuring motion invariants in state space. Numerical simulations showcase how nonlinear activation functions not only maintain limit cycles, but also enhance the numerical stability of the system integration process, mitigating those instabilities that are commonly associated with the forward Euler method. The experimental results of this analysis highlight practical considerations for designing neural architectures capable of capturing complex temporal dependencies, i.e., strategies for enhancing memorization skills in recurrent models.

Figures (5)

• Figure 1: Level sets $\{(x,y)\in\mathbb{R}^2:\log(\cosh(x)\cosh(y))=\ell\}$ of Eq. \ref{['eq:energy-2d']} with $\omega=1$ for $\ell=0.2,1.5,3$.
• Figure 2: State-space (2D). The different limit cycles associated with different activation functions. On the left column the linear case; in the center the tanh; on the right the hard-tanh. Each axis is a component of the state, the black dot and the blue star are the initial and final state, respectively. The state trajectory is enriched by a color gradient to suggest the evolution of the system through time, with violet points representing the state during the first steps of the simulation, and the yellow being the last ones, as also suggested by the bar on the right of the grid of images.
• Figure 3: State-space (2D). The role of non-odd activation functions like the sigmoid (center) and the ReLU (right). In the last case the simulation was interrupted before its end because the norm of the state exceeded the threshold, with the red cross marking the value of the state right before the interruption.
• Figure 4: Spectral response of skew-symmetric 2x2 RNN with selected angular frequencies (different colors). The y-axis is the amplitude (log-scale), while the x-axis is the frequency (Hz). The first two plots from the left refer to the linear and the tanh cases, respectively. Still proceeding left-to-right we can see the case of hard-tanh activation function for $w_m=10$ and $w_m=25$ respectively.
• Figure 5: Instability arising from the Euler's discretization (top left) is mitigated by increasing RNN with skew-symmetric matrices and tanh activation function ensures perpetual generation; however, issues arise with Euler's discretization (left). The neural response (top-row, x-axis: time, y-axis: output) is shown to increase as the number of neurons grows (the hidden dimensions are $n=\left\{2,20,40\right\}$ from left to right). As the dimension increases, numerical stability improves, and the spectrum becomes richer but remains concentrated at lower frequencies. The lower row is the spectrum of the generated signal computed with the Short-time FFT, with yellower points highlighting frequencies with bigger coefficients (x-axis: time, y-axis: frequency (Hz)).

Theorems & Definitions (8)

• Theorem 1
• Proposition 1
• Remark 1
• Theorem 2: Lyapunov Stability Theorem
• Remark 2: Linear RNN with skew-symmetric matrix
• Proposition 2
• Corollary 1
• Proposition 3