Neuronal correlations shape the scaling behavior of memory capacity and nonlinear computational capability of reservoir recurrent neural networks

TL;DR

The paper examines how memory capacity (MC) and nonlinear computational capability scale with the number of readout neurons $L$ in large reservoir RNNs. It introduces a Neumann-series–based analytic framework combined with the dynamical cavity method to account for neuronal correlations, showing MC grows sublinearly with $L$ under the scaling $L=\alpha\sqrt{N}$, with the decay tied to correlation strength. It further reveals that nonlinear processing powers emerge supralinearly in sequence as $L$ increases, across diverse dynamical regimes and architectures, guided by correlations rather than proximity to the edge of chaos. These results yield practical principles for designing scalable reservoir computing and provide new insights into how correlation structure shapes both linear memory and nonlinear computation in neural and artificial recurrent networks.

Abstract

Reservoir computing is a powerful framework for real-time information processing, characterized by its high computational ability and quick learning, with applications ranging from machine learning to biological systems. In this paper, we investigate how the computational ability of reservoir recurrent neural networks (RNNs) scales with an increasing number of readout neurons. First, we demonstrate that the memory capacity of a reservoir RNN scales sublinearly with the number of readout neurons. To elucidate this observation, we develop a theoretical framework for analytically deriving memory capacity that incorporates the effect of neuronal correlations, which have been ignored in prior theoretical work for analytical simplicity. Our theory successfully relates the sublinear scaling of memory capacity to the strength of neuronal correlations. Furthermore, we show this principle holds across diverse types of RNNs, even those beyond the direct applicability of our theory. Next, we numerically investigate the scaling behavior of nonlinear computational ability, which, alongside memory capacity, is crucial for overall computational performance. Our numerical simulations reveal that as memory capacity growth becomes sublinear, increasing the number of readout neurons successively enables nonlinear processing at progressively higher polynomial orders. Our theoretical framework suggests that neuronal correlations govern not only memory capacity but also the sequential growth of nonlinear computational capabilities. Our findings establish a foundation for designing scalable and cost-effective reservoir computing, providing novel insights into the interplay among neuronal correlations, linear memory, and nonlinear processing.

Figures (9)

• Figure 1: (a) Overview of RC with a random RNN receiving input signals and inherent neuronal noise. The shaded region represents the readout neurons. (b) Numerical simulations for memory capacity ($MC$) of the reservoir RNNs with network size $N=10000$, noise intensity $\sigma_n^2 = 0.1^2$, and activation function $\phi(x) = \tanh(x)$. Shaded area represents mean$\pm$std of direct numerical simulations for 10 different network and input signal realizations. The dashed line indicates the theoretical upper bound of memory capacity, i.e., $MC=L$. Each $MC$ curve shows a sublinear scaling with increasing $L$. In the simulations, the sum of $M_d$ is calculated up to $d=10^3$. Simulation time length is $T=10^4$, and a washout period of $T_{\rm washout} = 10^3$ steps is used to discard initial transients.
• Figure 2: Memory capacity as a function of $L$. An inset shows corresponding decay rate $r(L)$. Solid curves represent the analytical solutions given by Eqs.(\ref{['mc theory for erf']})(\ref{['DMFeq for erf']}), while error bars indicate mean$\pm$std for 10 networks and inputs realizations obtained through numerical simulations. Star marks represent the upper bounds of the sufficient conditions for the convergence of the series expansion. Dashed lines indicate divergence of theoretical values. The theoretical values and simulation results show excellent agreement in the region where the analytical solution converges. The activation function is an error function, $\phi(x) = \int_0^x e^{-\frac{\pi}{4} t^2}dt$. The input intensity is $\sigma_s^2 = 1.0^2 / \sqrt{N}$ ($\tilde{\sigma}_s^2 = 1.0^2$). The network size is $N=10000$, and simulation time length is $T=10^4$. The transient washout time is $T_{\rm washout}=10^3$.
• Figure 3: Analytical solutions for the decay rate of memory capacity as a function of $\alpha$. The activation function is an error function, $\phi(x) = \int_0^x e^{-\frac{\pi}{4} t^2}dt$. Each figure varies a single parameter: (a) input intensity $\tilde{\sigma}_s^2$, (b) recurrent weight scale $g$, (c) noise intensity $\sigma_n^2$, while the remaining parameters are fixed ($g=1.2$, $\tilde{\sigma}_s^2 = 1.0^2$, $\sigma_n^2=0.5^2$). A gradient from darker to lighter gray lines indicates a decreasing level of neuronal correlations. The decay rate for systems with strong neuronal correlations decreases more rapidly.
• Figure 4: Relationship between neuronal correlation and the half-life of memory capacity growth. The x-axis represents the level of neuronal correlations quantified by the root mean square (RMS) of pairwise Pearson correlation coefficients. The y-axis denotes $L_{\rm half}$, the definition of which is graphically shown in the inset of panel (a). (a) Results for RNNs with $\phi=\tanh$. Each data point corresponds to a set of hyperparameters sampled from $g\sim U(0.2, 3)$, $\sigma_s \sim U(0.1, 3)$ and $\sigma_n \sim U(0, 3)$. Red points indicate parameter regimes beyond our theoretical framework, as determined by $L_{\rm half}$ surpassing the deviation threshold of the theoretical values of $MC$ (Fig.\ref{['fig:analytical MC']}). (b) Results for RNNs with $\phi(x)= \max (0, x)$ (ReLU). Each data point corresponds to a set of hyperparameters sampled from $g\sim U(0.2, \sqrt{2})$, $\sigma_s \sim U(0.1, 3)$ and $\sigma_n \sim U(0, 3)$. For both (a) and (b), a total of 100 samples are plotted. Each data point represents the mean value over 10 network and input realizations. Error bars indicate the standard deviation for both $L_{\rm half}$ and $\sqrt{\langle \rho_{ij}^2 \rangle_{\rm pairwise}}$, but most of the error bars for the latter are too small to be visible behind the data points. Overall, for both activation functions, $L_{\rm half}$ tends to be smaller for stronger neuronal correlations. All simulations use a network size of $N=10000$ and a simulation time of $T=10^4$. The transient washout time is $T_{\rm washout}=10^3$.
• Figure 5: Numerical simulations of neuronal correlations and memory capacity of RNNs with recurrent weights drawn from a Cauchy distribution. (a) The level of neuronal correlations, quantified by the root mean square of pairwise Pearson correlation coefficients, as a function of the scale parameter $\gamma$. Data are shown for network sizes $N=1000,5000,$ and $10000$. (b) Left: Memory capacity (MC) as a function of the number of readout neurons, $L$. Right: Decay rate ($r(L)$) as a function of $L$. Both plots show data for different values of $\gamma$ with a fixed network size of $N=10000$. Networks with a large $\gamma$ exhibit weaker neuronal correlations and, consequently, a slower decay of $r(L)$. For both (a) and (b), the simulation time length is $T=10^4$. The input and noise intensity are $\sigma_s^2=1.0$ and $\sigma_n^2 = 0.0$, respectively. The activation function is $\phi=\tanh$. Errorbars and shaded areas represent mean$\pm$std over 10 network and input realizations.