Harnessing and modulating chaos to sample from neural generative models

TL;DR

The paper investigates how chaotic neural dynamics can be harnessed to sample from generative models, proposing two architectures that leverage chaos for probabilistic computation. One architecture uses a chaotic reservoir to generate latent variability that reshapes through a GAN, while the other combines structured memory with chaotic fluctuations in a mixed connectivity network, with a tunable gain $g$ controlling sampling speed and the balance between exploration and exploitation. Key contributions include demonstrating that population-level chaos provides rapid, scalable sampling and that gain modulation serves as a biologically plausible mechanism to adjust sampling rate. This work offers a bridge between chaotic dynamics, energy-based memory models, and modern generative modeling, with potential implications for neuroscience understanding and neuromorphic computing.

Abstract

Chaos is generic in strongly-coupled recurrent networks of model neurons, and thought to be an easily accessible dynamical regime in the brain. While neural chaos is typically seen as an impediment to robust computation, we show how such chaos might play a functional role in allowing the brain to learn and sample from generative models. We construct architectures that combine a classic model of neural chaos either with a canonical generative modeling architecture or with energy-based models of neural memory. We show that these architectures have appealing properties for sampling, including easy biologically-plausible control of sampling rates via overall gain modulation.

Figures (4)

• Figure 1: Chaotic dynamics in a randomly-connected recurrent network of neurons. (a) Top row shows time-varying network input to 3 neurons (i.e., $h$ in Eq. \ref{['eq:somp_chaos_net']}), with each neuron shown as a different trace. Bottom row shows corresponding activity or output from these 3 neurons (i.e., $\phi(h)$). Left to right show increasing values of gain, with $g=2, 4, 10$. Trajectories become increasingly chaotic as gain increases. (b) Sensitive dependence on initial conditions. Top and bottom panels show two different views of a network started at very similar initial conditions. Top panel shows traces over time, with solid traces corresponding to original initial condition and dashed traces corresponding to same neurons in perturbed initial conditions. Bottom panel shows 3D state space plot of trajectories, with blue and red corresponding to original and perturbed initial conditions respectively, and network dynamics shown only for the first 20 units of time, with start and end points of the trajectories indicated. The perturbation to the initial condition of each neuron is uniformly drawn from $[0, 10^{-4}]$ and network gain, $g=10$. The original and slightly perturbed trajectories rapidly diverge and become uncorrelated, as can be seen from the separation between the solid and dashed traces at the top and between red and blue endpoints at the bottom. Network size $N=250$ in all plots.
• Figure 2: Using chaos to sample from a Generative Adversarial Network. (a) Schematic of the generator. The recurrent network shown in the big circle has randomly-chosen recurrent connections (dark blue) and is in the chaotic regime. Projections out from the recurrent network are sparse and random, forming an expander graph (light blue). This activity is then propagated through a multilayer convolutional network to generate images. Blue connections are random and fixed during training, while black connections are trained. The architecture also includes a discriminator, which is not shown for simplicity. (b) Samples from the GAN trained on the CelebA dataset liu2015faceattributes.
• Figure 3: Controlling the speed of sampling by modulating the gain. (a) Consecutive samples from the network in Fig. \ref{['fig:chaos_gen']} at intervals of $0.15 \tau$ with gain set to $2$. (b) Consecutive samples from the network in Fig. \ref{['fig:chaos_gen']} at intervals of $0.15 \tau$ with gain set to $8$. Note that the network was trained with gain set to $4$ and not retrained to generate these samples.
• Figure 4: Recurrent networks with combined structured and random connectivity. (a) Schematic of network with a combination of symmetric low-rank connectivity (black) and high-dimensional random connectivity (blue). (b) Schematic of energy-based dynamics in network when only symmetric low-rank component is present (i.e., gain of random connectivity is set to $0$). Dynamics moves the network state downhill in energy until it converges to an attractor state. Panels c, d apply to networks whose connectivity is given by $W = W_{\text{struct}} + gW_\text{rand}$, for some fixed $g$. Columns left to right show $g=0, 3, 5,$ and $8$. Note that the plot for each gain uses a different network initial condition and thus is not expected to converge to the same attractor state. (c) Top and bottom panels respectively show time-varying network input to and output from 3 neurons (i.e., as in Fig. 1a except network combines random and structured connectivity). Note the convergence to the stable attractor state in the first panel (when $g=0$). (d) Top panels show projection (normalized dot product) of network activity vector onto the $10$ patterns stored in the structured connectivity (i.e., the minima of the corresponding energy function), with each row showing a different projection. Bottom panels show projection of network activity onto $190$ orthogonal directions. For low gain, the activity is confined to the subspace spanned by stored patterns. As gain increases (i.e., left to right), network activity spends more time exploring orthogonal directions. (e) Results from network whose connectivity is given by $W = W_{\text{struct}} + g(t)W_\text{rand}$, where $g(t)$ cycles between a low value ($g=1$) and a high value ($g=8$). As in panel (d), top and bottom panels show projection onto stored patterns and orthogonal directions respectively. When $g$ is low, the network settles into one of the energy minima (seen as bright bands in top panel and dark bands in bottom panel). When $g$ is high, the network explores state space. Cycling $g$ allows the network to explore multiple energy minima. Note that the scale in the top panel is different, to highlight the sharp convergence to the energy minimum (i.e., overlap is close to perfect).