Slow Transition to Low-Dimensional Chaos in Heavy-Tailed Recurrent Neural Networks

TL;DR

This paper investigates how heavy-tailed synaptic weights, modeled by Lévy $\alpha$-stable distributions, shape the dynamics of finite-size recurrent neural networks. Using a QR-based Lyapunov analysis and two dimensionality metrics, it shows a finite-size quiescent-to-chaotic transition at a gain $g^*$ that shifts with network size $N$ and tail index $\alpha$, in contrast to infinite-width mean-field predictions of ubiquitous chaos. Heavier tails yield a slower, more robust approach to chaos, broadening the edge-of-chaos regime, but simultaneously compress the chaotic attractor into a lower-dimensional slow manifold, reducing $D_{KY}$ and PR. This robustness-versus-dimensionality tradeoff aligns with biological connectivity statistics and has implications for reservoir computing and brain-like information processing, offering a tractable finite-size framework for heavy-tailed neural circuits. Overall, the work provides theoretical and computational tools to understand how realistic, finite neural networks manage stability, richness of dynamics, and computational capacity under heavy-tailed connectivity.

Abstract

Growing evidence suggests that synaptic weights in the brain follow heavy-tailed distributions, yet most theoretical analyses of recurrent neural networks (RNNs) assume Gaussian connectivity. We systematically study the activity of RNNs with random weights drawn from biologically plausible Lévy alpha-stable distributions. While mean-field theory for the infinite system predicts that the quiescent state is always unstable -- implying ubiquitous chaos -- our finite-size analysis reveals a sharp transition between quiescent and chaotic dynamics. We theoretically predict the gain at which the system transitions from quiescent to chaotic dynamics, and validate it through simulations. Compared to Gaussian networks, heavy-tailed RNNs exhibit a broader parameter regime near the edge of chaos, namely a slow transition to chaos. However, this robustness comes with a tradeoff: heavier tails reduce the Lyapunov dimension of the attractor, indicating lower effective dimensionality. Our results reveal a biologically aligned tradeoff between the robustness of dynamics near the edge of chaos and the richness of high-dimensional neural activity. By analytically characterizing the transition point in finite-size networks -- where mean-field theory breaks down -- we provide a tractable framework for understanding dynamics in realistically sized, heavy-tailed neural circuits.

Figures (21)

• Figure 1: (A): Transition point $g^*$ predicted by our theory as a function of network size for various $\alpha$. The transition point of Gaussian networks rapidly converges to the mean-field limit (dashed line). In contrast, the transition point of heavy-tailed networks decays slowly towards zero. (B): The fraction of small ($\epsilon=0.1$) final state components in linear networks with $\alpha=1$ and $N=3000$ evolved for $T=100$ steps from random initial conditions as a function of $g$. In the annealed case, we observe a sharp transition at the location predicted by the theory. In the quenched case, each individual realization exhibits a sharp transition (thin blue lines), but its location varies between different realizations of the weight matrix. Thus, when averaged over the realizations (thick blue line and dots; shaded region shows the $\pm3$ standard error), the transition looks smoother than in the annealed case. Nonetheless, its location is approximately predicted by the theory. (C): Same as B but with $N=100$. As predicted by the theory, the transition point shifts to the right with decreasing $N$. Moreover, the location of the transition in the quenched case varies more in smaller networks.
• Figure 2: Maximum Lyapunov exponent ($\lambda_{\text{max}}$) as a function of gain $g$ for autonomous recurrent networks with different tail indices $\alpha$, shown for: (A)$N = 1000$, (B)$N = 3000$, and (C)$N = 10000$. Curves show mean across 10 trials; shaded regions denote $\pm$1 SD. We let the networks evolve for $T=3000$ steps, among which the Lyapunov exponents are accumulated over the last $K=100$ steps. See results under noisy stimulus and ablation studies in Appendices \ref{['app:noisy']}, \ref{['app:robust_mle']}. Heavier-tailed networks (lower $\alpha$) exhibit a slower, more gradual increase in $\lambda_{\text{max}}$ near the transition (where $\lambda_{\text{max}} = 0$), resulting in a broader edge-of-chaos regime with respect to $g$. Dashed lines and legend mark the average critical gain $g^*$ at which $\lambda_{\text{max}}$ first crosses zero. As $N$ increases, this transition shifts leftward, especially for lower $\alpha$, in line with our theoretical predictions on finite-size effects.
• Figure 3: Heavy-tailed networks exhibit lower-dimensional attractors near the edge of chaos. Curves show mean across 10 trials for networks of size $N=1000$; shaded regions denote $\pm$1 SD. See Appendix \ref{['app:noisy']} for results under noisy stimuli. The implementation details and ablation studies are provided in Appendices \ref{['app:robust_LEs']}, \ref{['app:robust_dimension']}. (A) Distributions of top $100$ Lyapunov exponents for varying $\alpha$ show fewer exponents near zero in heavier-tailed networks at estimated $\langle g^* \rangle$ obtained in Fig \ref{['fig:MLE']}A, indicating a lower-dimensional slow manifold. x-axis truncated at the left to omit near-zero tails for clarity. (B) The Lyapunov dimension is smaller for heavier-tailed networks near the regime of edge of chaos, reflecting fewer directions of local expansion in phase space. (C) The participation ratio dimension is similarly smaller with lower $\alpha$ near the edge of chaos, showing reduced variance homogeneity across neural modes. Together, these results indicate that while heavy-tailed networks maintain robustness to neural gain near chaos, they compress dynamics into a lower-dimensional attractor.
• Figure 4: Same as Fig. \ref{['fig:MLE']}, but with an addition of $\alpha=0.5$.
• Figure 5: Effect of network size under small i.i.d. noisy input. Maximum Lyapunov exponent ($\lambda_{\max}$) as a function of gain $g$ in noisy stimulus-driven recurrent networks with Lévy $\alpha$-stable weight distributions. Curves show mean across 10 trials; shaded regions denote $\pm$1 SD. Each panel corresponds to a different network size: (A) $N = 1000$, (B) $N = 3000$, and (C) $N = 10000$. Curves show mean across 3 trials; shaded regions denote $\pm$1 SD. As in the autonomous case, if a transition exists, then heavier-tailed networks exhibit a slower transition and wider critical regime near $\lambda_{\max} = 0$. The critical gain $g^*$ (dashed line) shifts leftward with increasing $N$, consistent with finite-size theory.