Kinetic energy in random recurrent neural networks

TL;DR

The paper investigates how kinetic energy characterizes chaotic dynamics in random recurrent neural networks (RNNs) by applying dynamical mean-field theory (DMFT) to a network with Gaussian weights $J_{ij}\sim \mathcal{N}(0,g^2/N)$. It derives a self-consistent description of the stationary kinetic energy $\Gamma_0$ via an effective potential $V(\Delta;\Delta_0)$ and shows that $\Gamma_0$ turns on continuously at the chaos transition $g_c=1$ with cubic scaling $\Gamma_0 \sim \frac{1}{3}(g-1)^3$; near criticality, the equal-time variance scales as $\Delta_0 \sim (g-1) + \tfrac{7}{6}(g-1)^2$. The study also maps the stationary RNN fluctuations to a gradient dynamics at an effective temperature $T_{\mathrm{eff}}$, finding agreement in one-time activity distributions and establishing a direct link between arc-length growth and $\sqrt{\Gamma_0}$; finite-size effects reveal geometric distinctions in phase space between the two dynamics. Overall, the work provides a kinetic-energy–driven framework to understand the chaos landscape in high-dimensional neural systems, with implications for reservoir computing and learning.

Abstract

High-dimensional chaotic dynamics can emerge in a large random recurrent neural network when the synaptic gain crosses a threshold. Recent works showed that the kinetic energy of neural activity links the chaotic dynamics and the supporting unstable fixed points (equilibria) in the phase space. Here, we investigate the kinetic-energy-centric properties of random recurrent neural networks by combining dynamical mean-field theory with extensive numerical simulations. We find that the average kinetic energy shifts continuously from zero to a positive value at a critical value of coupling variance (synaptic gain) and exhibits a cubic scaling behavior near the critical point from above. This scaling behavior is supported by numerical simulations and provides a quantitative characterization of how fast the dynamics change during the onset of chaos. The steady-state activity distribution is further calculated by the theory and compared with simulations on finite-size systems from the kinetic-energy optimization perspective as well. The activity distribution is also analyzed in a geometric angle, establishing a relationship between the original chaotic dynamics and the gradient dynamics of the kinetic energy. The trajectory length on the chaotic manifold can be derived from the stationary kinetic energy, and the associated stationary behavior is analyzed as well. This study provides a kinetic-energy-centric route toward understanding the dynamics landscape of recurrent neural networks, which may provide insights for reservoir computing and even for internal synaptic learning.

Figures (8)

• Figure 1: Sketch of network connectivity and dynamics. (a) Schematic illustration of the RNN connectivity structure. (b) Sample trajectories of five neurons for coupling strengths $g = 0.5,1.05,2$ in a neural pool of $N=5\,000$.
• Figure 2: The average kinetic energy of the system versus synaptic gain $g$. (a) Results for $0.8 \leq g \leq 1.5$; (b) An enlarged view of the critical regime $0.9 \leq g \leq 1.2$. Simulations were performed using networks of size $N=1000$, $5000$, and $10000$, with standard deviations computed from $100$ independent realizations. The DMFT result is obtained by numerically solving $\mathcal{F}(\Delta_0; \Delta_0) = 0$ for $\Delta_0$, and substituting the solution into Eq. (\ref{['Gamma_0']}). Standard deviations for the DMFT prediction are computed from $100$ numerical evaluations (see a detailed explanation in the main text).
• Figure 3: Comparison between the theoretical prediction [Eq. \ref{['Delta0_expan']} or Eq. \ref{['gamma0exp']}] in the limit $g \to 1^+$ and the numerical results from DMFT [solutions of $\mathcal{F}(\Delta_0;\Delta_0)=0$ or Eq. (\ref{['Gamma_0']})]. (a) The dependence of $\Delta_0$ on $\sigma$; (b) $\Gamma_0$ obtained by substituting $\Delta_0$ into the analytical expression [Eq. \ref{['gamma0asyt']}]. The DMFT shaded error is estimated from $200$ independent numerical evaluations via Eq. (\ref{['Gamma_0']}). (c) Enlarged view of the comparison close to the critical point in log-log scale. Note that the mean value of $\Delta_0$ is inserted into Eq. \ref{['gamma0asyt']}.
• Figure 4: Probability distribution of stationary activity. (a) Distribution of $x(t)$ at steady states for different coupling strengths $g=1.1,1.3,1.5$. Histograms represent numerical simulation results from RNN simulations ($N=5000$), while dashed lines correspond to the Gaussian distributions obtained by DMFT. (b) Mean kinetic energy as a function of temperature ($g=1.5,N=5000$). The blue solid curve represents the steady-state average kinetic energy of the gradient descent (GD) dynamics in Eq. \ref{['GD_dynamics']}. The purple dashed curve shows the corresponding steady-state average for the RNN dynamics in Eq. \ref{['neuron_dynamics']} with the same interaction matrix. The vertical dashed line indicates the effective temperature (mean matched with the RNN counterpart). The mean kinetic energy and the associated error bars (only shown for the matched point) are computed from $200$ statistically independent samples taken in the stationary regime. (c) Comparison between sampled steady state activities of the two dynamics (the GD one at the effective temperature) ($g=1.5,N=5000$).
• Figure 5: Dynamical trajectories of the RNN (network size $5000$) and the corresponding trajectories from the DMFT equations. Parameters from left to right: $g=1.1,1.3,1.5$.