Phase transitions from linear to nonlinear information processing in neural networks

TL;DR

The authors demonstrate a phase-transition-like jump in information processing capacity when reservoir networks transition from linear to nonlinear regimes as input nonlinearity increases, controlled by the input-weight standard deviation and modulated by noise. Using echo state networks with fixed recurrent weights and linear readouts, they quantify capacity via $r^2$ and reveal a sharp, possibly discontinuous boundary between linear and nonlinear processing that strengthens with larger $N$. A universal scaling emerges: the transition point obeys $\sigma_{\text{in},0} \propto \sigma_{\xi}^{1/3}$, and the capacity curves collapse when using the rescaled input $\sigma_{\text{in}}/\sigma_{\xi}^{1/3}$; the exponent $1/3$ reflects the leading nonlinearity order near zero. Network-size analyses show the transition sharpens with $N$, with $\Delta r^2_{\text{max}} \propto N^{\alpha}$ and $\tilde{\sigma}_{\text{in},0} \propto N^{-\beta}$, where $\alpha>\beta$, hinting at a discontinuity in the $N\to\infty$ limit. The results hold across tasks (XOR-based, NARMA, Lorenz, and delay) and suggest a broader class of nonequilibrium phase transitions in high-dimensional dynamical systems, though a full theoretical explanation remains for future work.

Abstract

We investigate a phase transition from linear to nonlinear information processing in echo state networks, a widely used framework in reservoir computing. The network consists of randomly connected recurrent nodes perturbed by a noise and the output is obtained through linear regression on the network states. By varying the standard deviation of the input weights, we systematically control the nonlinearity of the network. For small input standard deviations, the network operates in an approximately linear regime, resulting in limited information processing capacity. However, beyond a critical threshold, the capacity increases rapidly, and this increase becomes sharper as the network size grows. Our results indicate the presence of a discontinuous transition in the limit of infinitely many nodes. This transition is fundamentally different from the conventional order-to-chaos transition in neural networks, which typically leads to a loss of long-term predictability and a decline in the information processing capacity. Furthermore, we establish a scaling law relating the critical nonlinearity to the noise intensity, which implies that the critical nonlinearity vanishes in the absence of noise.

Figures (10)

• Figure 1: Schematic illustration of the information processing capacity as a function of system nonlinearity. (a) For a finite number of degrees of freedom, the capacity increases gradually with weak nonlinearity and decreases rapidly when strong nonlinearity induces chaos. The question arises as to what happens when the number of degrees of freedom goes to infinity. As illustrated in panel (b), we demonstrate a possibility that the capacity undergoes a discontinuous jump at a critical point, indicated by the red arrow.
• Figure 2: Information processing capacity $r^2$ and its fluctuations $\sigma(r^2)$ for an echo state network as a function of the input standard deviation $\sigma_{\text{in}}$ and the spectral radius $\rho$ of the recurrent weight matrix. Panels (a)-(c) show the averaged capacity $r^2$ for different numbers of nodes, $N=200$, $500$, and $1000$. The blue solid curves indicate the boundary of the order-to-chaos transition, determined by the Lyapunov exponent. Panel (d) shows similar data for $\rho=0.8$, which is highlighted by the green solid lines in (a)-(c). We identify four regimes (1)-(4) in (b). Regimes (1) and (2) correspond to linear and nonlinear information processing, respectively, and the boundary between these two regimes becomes sharper as $N$ increases. Panels (e)-(g) show the logarithm of the standard deviation $\sigma(r^2)$ of the capacity over different network realizations, while panel (h) shows $\sigma(r^2)$ for $\rho=0.8$. At the transition between regimes (1) and (2), $\sigma(r^2)$ exhibits a pronounced peak, signaling a phase transition. For all data, the noise intensity is $\sigma_{\xi}=10^{-8}$ and the delay for the XOR task is set to $k=10$.
• Figure 3: Scaling of the transition point with respect to the noise intensity $\sigma_{\xi}$. (a) Information processing capacity $r^2$ as a function of the input standard deviation $\sigma_{\text{in}}$ for various noise intensities $\sigma_{\xi}=10^{-10}$, $10^{-9}$, $10^{-8}$, $10^{-7}$, $10^{-6}$. The network has $N=1000$ nodes, and the spectral radius of $W$ is $\rho=0.8$. The XOR task uses a delay of $k=10$. (b) $r^2$ plotted against the rescaled input standard deviation $\sigma_{\text{in}}/\sigma_{\xi}^{1/3}$. The collapse of the curves for different $\sigma_{\xi}$ onto a single master curve is clearly observed. (c) Example of fitting $r^2$ versus $\sigma_{\text{in}}$ for $\sigma_{\xi}=10^{-8}$ using a logarithmic logistic function. The dashed line indicates the transition point $\sigma_{\text{in}, 0}$, defined as the point where the derivative $dr^2/d\sigma_{\text{in}}$ attains its maximum. (d) $\sigma_{\text{in}, 0}$ as a function of the noise intensity $\sigma_{\xi}$ for different spectral radii $\rho = 0.4$, $0.6$, $0.8$. The dashed lines show power-law fits of the form $\sigma_{\text{in}, 0} \propto \sigma_{\xi}^\eta$, with $\eta=0.340$ for $\rho=0.4$, $\eta=0.334$ for $\rho=0.6$, and $\eta=0.333$ for $\rho=0.8$.
• Figure 4: Network size scaling of the transition point. (a) Information processing capacity $r^2$ versus the rescaled input standard deviation $\tilde{\sigma}_{\text{in}}=\sigma_{\text{in}}/\sigma_{\xi}^{1/3}$ for $\sigma_{\xi}=10^{-8}$ at several network sizes $N$ (XOR task with delay $k=20$). The dashed curves show logarithmic-logistic fits. The peak value of $r^2$ saturates to $1$ for $N \gtrsim 2000$, beyond which both $\Delta r^2_{\text{max}}$ and $\tilde{\sigma}_{\text{in},0}$ exhibit power-law scaling with respect to $N$. (b) Network size dependence of the maximum derivative $\Delta r^2_{\text{max}}$ at the transition point in a double-log plot. The dashed lines represent $\Delta r^2_{\text{max}} \propto N^{\alpha}$. (c) Network size dependence of the transition point $\tilde{\sigma}_{\text{in}, 0}$ in a double-log plot. The dashed lines represent $\tilde{\sigma}_{\text{in}, 0} \propto N^{-\beta}$. (d) Exponents $\alpha$ and $\beta$ for different delay values $k$. The power-law fits for $\tilde{\sigma}_{\text{in}, 0}$ and $\Delta r^2_{\text{max}}$ are performed for $N \geq 2200$. The error bars indicate the standard errors from the fitting. The fact that $\alpha > \beta$ implies that the transition sharpens faster than $\tilde{\sigma}_{\text{in}, 0}$ vanishes.
• Figure 5: Information processing capacity $r^2$ for the NARMA, Lorenz, and delay tasks. (a) $r^2$ for the NARMA task as a function of $\sigma_{\text{in}}$ and $\rho$ with $N=500$. (b) $r^2$ for the NARMA task with $\rho=0.8$ and various values of $N$. (c) $r^2$ for the Lorenz task as a function of $\sigma_{\text{in}}$ and $\rho$ with $N=500$. (d) $r^2$ for the Lorenz task with $\rho=0.8$ and various values of $N$. (e) $r^2$ for the delay task as a function of $\sigma_{\text{in}}$ and $\rho$ with $N=500$. The value of delay is $k=30$. (f) $r^2$ for the delay task with $\rho=0.8$ and various values of $N$. The regions labeled (1) and (2) indicate the linear and nonlinear information processing phases, respectively. Panels (b), (d), and (f) show that the boundary between phases (1) and (2) becomes shaper as $N$ increases.