Connectivity structure and dynamics of nonlinear recurrent neural networks

TL;DR

The paper addresses how structured connectivity shapes high-dimensional dynamics in nonlinear recurrent networks, motivated by connectomic data. It introduces the random-mode model, a SVD-like, extensive-rank generative framework, and develops a path-integral DMFT to derive two-point and four-point statistics that capture population activity and its dimensionality. The key finding is that the dimension of activity and its timescales depend primarily on the effective coupling strength $g_{ ext{eff}}$ and the effective rank $ ext{PR}^D$, with structured left-right overlaps (as observed in the Drosophila connectome) introducing additional dependencies on the full joint distribution. Single-neuron heterogeneity and mode overlaps can meaningfully modulate population dimensionality even when single-neuron activity appears uninformative, offering concrete predictions for z-scored data and experimental analyses. Overall, the framework links connectome statistics to functional population dynamics, providing a tractable path to incorporate extensive spectral structure into theories of neural computation and learning.

Abstract

Studies of the dynamics of nonlinear recurrent neural networks often assume independent and identically distributed couplings, but large-scale connectomics data indicate that biological neural circuits exhibit markedly different connectivity properties. These include rapidly decaying singular-value spectra and structured singular-vector overlaps. Here, we develop a theory to analyze how these forms of structure shape high-dimensional collective activity in nonlinear recurrent neural networks. We first introduce the random-mode model, a random-matrix ensemble related to the singular-value decomposition that enables control over the spectrum and right-left mode overlaps. Then, using a novel path-integral calculation, we derive analytical expressions that reveal how connectivity structure affects features of collective dynamics: the dimension of activity, which quantifies the number of high-variance collective-activity fluctuations, and the temporal correlations that characterize the timescales of these fluctuations. We show that connectivity structure can be invisible in single-neuron activities while dramatically shaping collective activity. Furthermore, despite the nonlinear, high-dimensional nature of these networks, the dimension of activity depends on just two connectivity parameters -- the variance of the couplings and the effective rank of the coupling matrix, which quantifies the number of dominant rank-one connectivity components. We contrast the effects of single-neuron heterogeneity and low dimensional connectivity, making predictions about how z-scoring data affects the dimension of activity. Finally, we demonstrate the presence of structured overlaps between left and right modes in the Drosophila connectome, incorporate them into the theory, and show how they further shape collective dynamics.

Figures (11)

• Figure 1: Analysis of Drosophila central-brain connectome. (a) Volume of fly brain for which reconstruction was performed (blue; reproduced from scheffer2020connectome). (b) Normalized coupling matrix (elements summed within $10 \times 10$ blocks to aid visualization). (c) Singular-value spectra of the normalized coupling matrix (red) and an i.i.d. random matrix (gray). The fly connectome exhibits a smooth spectrum that decays quickly, corresponding to a reduced participation ratio. $N = 18028$ neurons.
• Figure 2: Schematic of the random-mode model. Upper: couplings $\bm{J}$ are generated as a sum of outer products, ${\bm \ell}_a {\bm r}^T_a$, with component strengths $D_a$. Lower: the two-point function $C^\phi_\star(\tau)$ and four-point function $\Psi^\phi_\star(\tau)$ are calculated in terms of the statistics of $D_a$. The two-point function depends only on the effective gain $g_\text{eff}$, while the four-point function depends on both $g_\text{eff}$ and $\text{PR}^D$, the effective dimension of the connectivity determined by the $D_a$ distribution.
• Figure 3: Duality of neuron-by-neuron and time-by-time covariances and its relation to the dimension of activity. Both plots are based on the same simulation of a network of $N=2500$ neurons with $g=2.25$ and i.i.d. couplings. The dimension can be computed either by computing the statistics of the off diagonals of the neuron-by-neuron covariance $C^\phi_{ij}(0)$ (left), or by computing the fluctuations away from the translation-invariant mean-field form of the time-by-time covariance $C^\phi(t_1,t_2)$ (right).
• Figure 4: Dimension of activity and collective timescales in the random-mode model. (a) Activity dimension $\text{PR}^\phi$ versus effective rank $\alpha \text{PR}^D$ for various coupling strengths $g_\text{eff}$. Thin dots, individual simulations; thick dots, means over ten simulations; lines, theoretical predictions. Inset: extended $g_\text{eff} = 6$ case, showing convergence to i.i.d. coupling behavior with growing $\alpha$. (b) Normalized four-point function $\overline{\Psi^{\phi}(\tau, 0)} \equiv \Psi^{\phi}(\tau, 0)/\Psi^{\phi}(0, 0)$ for various $g_\text{eff}$ and $\alpha \text{PR}^D$. Inset: theory curves for all $\alpha \text{PR}^D$, demonstrating relative invariance of collective timescales to the effective rank. Simulations use $N=5000$ neurons.
• Figure 5: Effect of single-neuron heterogeneity on dimension of activity in the random-mode model. (a) Dimension of normalized activity $\text{PR}^\phi$ versus participation ratio of gain distribution $\text{PR}^G$ for various coupling strengths $g_\text{eff}$ and effective ranks $\alpha \text{PR}^D$. Thin dots, individual simulations; thick dots, means over ten simulations; lines, theoretical predictions. (b) Fractional reduction in dimension for weighted activations relative to $\text{PR}^\phi$. Blue, $G_i \phi_i$; purple, $G^\text{readout}_i \phi_i$. $g_\text{eff} = 10$. Thick dots, means over ten simulations, averaging before taking the ratio; lines, theoretical predictions. All simulations use $N=5000$ neurons.