Introduction to dynamical mean-field theory of randomly connected neural networks with bidirectionally correlated couplings

TL;DR

A pedagogical introduction of this method in a particular example of random neural networks, where neurons are randomly and fully connected by correlated synapses and therefore the network exhibits rich emergent collective dynamics.

Abstract

Dynamical mean-field theory is a powerful physics tool used to analyze the typical behavior of neural networks, where neurons can be recurrently connected, or multiple layers of neurons can be stacked. However, it is not easy for beginners to access the essence of this tool and the underlying physics. Here, we give a pedagogical introduction of this method in a particular example of random neural networks, where neurons are randomly and fully connected by correlated synapses and therefore the network exhibits rich emergent collective dynamics. We also review related past and recent important works applying this tool. In addition, a physically transparent and alternative method, namely the dynamical cavity method, is also introduced to derive exactly the same results. The numerical implementation of solving the integro-differential mean-field equations is also detailed, with an illustration of exploring the fluctuation dissipation theorem.

Figures (5)

• Figure 1: Comparison between observables obtained from direct simulation (color curves) and iterative mean-field solution (dash curves). The parameters set $\{g, \eta, \sigma, \phi, \Delta t, t^{\prime}\}$ is $\{0.2, 0.5, 0.1, \tanh, 0.1(ms), 10(ms)\}$. Inset: relative differences vs $N$.
• Figure 2: Convergence of observables to analytic fixed point solutions during iteration of the DMFT equation. The parameters set $\{g, \eta, \sigma, \Delta t\}$ is $\{0.2, 0.5, 0.1, 0.1(ms)\}$. (a) Transfer function $\phi = \tanh$. (b) Transfer function $\phi = \mathrm{ReLU}$.
• Figure 3: Effective temperature of the system given different asymmetry correlation levels and nonlinearities. The waiting time $t^{\prime}$ is fixed at $9$ms and the color of points becomes lighter as $t$ increases. The dash line indicates FDT for the linear system with symmetric connection $\eta=1$, whose thermodynamic temperature is $T=\sigma^2/2=0.5$ (indicated by a black circle in the inset). (a) Comparison among different asymmetry correlation levels in the linear system. For $\eta=[-1,0,1]$, the effective temperatures obtained by a linear fitting are $T_{\mathrm{eff}} = [0.541, 0.531, 0.514]$, respectively (inset). (b) Comparison among different nonlinear functions when $\eta=1$. For $\phi$ selected to be ReLU, Tanh and linear, the effective temperatures obtained by a linear fitting are $T_{\mathrm{eff}} = [0.516, 0.515, 0.514]$, respectively (inset).
• Figure 4: The estimated effective temperature versus the waiting time. Simulation parameters are the same as in Fig. \ref{['fig:fdt']}.
• Figure 5: Response and correlation functions depend only on the time difference. Linear dynamics with $\eta=1$ is considered for an example. Simulation parameters are the same as in Fig. \ref{['fig:teff']}.