Learn to integrate parts for whole through correlated neural variability

TL;DR

The paper proposes a covariance-based computation framework in which perceptual information is embedded in the correlated variability of sensory neurons and transformed into downstream firing rates via a nonlinear moment mapping. Using a moment neural network (MNN) as a bridge between sensory covariance and readout activity, the authors show that motion direction can be encoded in neural covariance and decoded with high fidelity, and that training under this framework enhances natural image classification both in accuracy and inference speed. They validate the approach with leaky integrate-and-fire neuron models and SPiking Neural Networks, perform an information-theoretic decomposition to show direction information largely resides in mean readouts, and extend the method to a complex visual task, highlighting the functional role of covariance beyond a secondary coding factor. These results suggest a hierarchical, covariance-driven processing scheme wherein perceptual information shifts from population covariance to individual neuron firing rates, with potential implications for brain-inspired learning and efficient SNN implementations.

Abstract

Sensory perception originates from the responses of sensory neurons, which react to a collection of sensory signals linked to various physical attributes of a singular perceptual object. Unraveling how the brain extracts perceptual information from these neuronal responses is a pivotal challenge in both computational neuroscience and machine learning. Here we introduce a statistical mechanical theory, where perceptual information is first encoded in the correlated variability of sensory neurons and then reformatted into the firing rates of downstream neurons. Applying this theory, we illustrate the encoding of motion direction using neural covariance and demonstrate high-fidelity direction recovery by spiking neural networks. Networks trained under this theory also show enhanced performance in classifying natural images, achieving higher accuracy and faster inference speed. Our results challenge the traditional view of neural covariance as a secondary factor in neural coding, highlighting its potential influence on brain function.

Figures (5)

• Figure 1: Encode perceptual information in the spatial-temporal covariance of sensory neurons. a, A schematic diagram showing how a stimulus $s$ is represented by the correlated or anticorrelated appearance of two components $c_1$ and $c_2$. The intensity of each component reaching sensory neurons fluctuates, leading to variable neural activities. The downstream network estimates $\hat{s}$ based on the responses of sensory neurons. b, A detailed examination of sensory neurons' activities reveals that the mean and covariance over short timescales capture the varying intensities of the stimulus components. The representation of stimulus $s$ encoded in the sign of the correlation between sensory neurons $N_1$ and $N_2$, responsive to $c_1$ and $c_2$ respectively.
• Figure 1: Analysis of noise correlation among the hidden neurons under the high contrast condition.a. Distribution of noise correlation $\rho_{ij}$ for pairs of hidden neurons ($i\neq j$) plotted against the difference in their preferred directions ($\Delta\theta_{ij}^{\rm prefer} = \theta_i^{\rm prefer} - \theta_j^{\rm prefer}$). The color bar represents the joint probability of $\Delta\theta_{ij}^{\rm prefer}$ and $\rho_{ij}$. b. Average noise correlation among hidden neurons as motion directions vary. c. Correlation patterns in the trained connection strengths $W_{\rm in}$ from all sensory neurons to individual hidden neurons. In both b and c, hidden neuron indices are sorted based on their preferred directions, ranging from $-\pi$ to $\pi$, and color bars indicate the strength of correlation coefficients. All results are obtained at the high contrast level where $c = 0.8$.
• Figure 2: Learn to infer motion direction through correlated neural variability.a, Task overview: Neurons arranged in a hexagonal grid collectively respond to the intensity of light and the corresponding rate of change of a moving grating. The downstream network learns to have a better estimate by minimizing readout errors and trial-to-trial variability. b, Evolution of validation loss across training epochs. c, Correlation pattern of the trained weights $W_{\rm in}$ for each sensory neuron projecting to hidden neurons. Indexes 1-527 denote intensity detectors, while the remaining represent change detectors. d, Visualization of the trained linear decoder $W_{\rm out}$, where the X and Y coordinates of each dot are the connection strengths that map the response of each hidden neuron to the 2D readout space. e, Normalized readout results for various motion directions at different contrast levels ($c \in [0.25, 0.5, 0.75]$). Gray arrows represent ground truths, while colored dots and ellipses illustrate readout mean and covariance, respectively. f, Average readout error (blue) and readout variability (orange) as a function of contrast. g-h, Tuning curves and Fano factor curves of hidden neurons with specific preferred motion directions (indicated by color, as in e). Transparency corresponds to contrast levels (higher contrast results in lower transparency).
• Figure 3: Dynamics of SNN in integrating local information for decoding directions. a, Raster plots that represent an intensity detector and a change detector at the same spatial location, alongside hidden neurons of the SNN under a moving grating with a direction of $\theta = 0.06$ rad. The hidden neurons are arranged based on their preferred directions. b, Readout error of the SNN as a function of readout time $\Delta t$. The inset provides a detailed view of the initial 100 ms of the readout error curve. c, Power spectral density and autocorrelation (inset) of the relative deviation of population spike count (1 ms time window). Black triangles mark the temporal frequency of the stimulus, including double and triple frequencies. d, Unwrapped average phase and average Kuramoto order parameter (inset) of sensory neurons' firing rates and hidden neurons' membrane potentials from 100 to 220 ms after stimulus onset. e, Quantification of direction information in the readout. The gray dotted line represents the theoretical bound of mutual information, specifically the entropy of moving direction $H(\theta)$. Components: tot - total mutual information between stimuli and readouts; lin - the sum of mutual information from individual neuron responses; sigsim - the difference in entropy between population and individual responses; cor - information from correlated neuron activities. In b, c, d, solid lines and shaded regions (with an error bar in the inset of d) depict the mean and standard deviation over 500 trials, averaged across 50 motion directions.
• Figure 4: Incorporating second-order information improves model performance in natural image classificationa, Task schematic: A pre-trained convolutional neural network (CNN) serves as the sensory system, extracting diverse features from the input image across multiple channels. These spatial features are then transformed into responses in the temporal domain, and their mean and covariance are computed using equation (\ref{['eq:theory_define_mean']})-(\ref{['eq:theory_define_cov']}). The downstream network utilizes the mean and covariance information to infer the image's category. b, Distribution of correlation coefficients in the covariance matrix obtained from all images in the dataset. c, Comparison of the classification accuracy of MNNs (With corr and Without corr) and an ANN after training. The error bars represent the standard deviation of 5 trials. d, The average probability of the correct prediction of the SNNs as a function of readout time. With corr, an MNN trained with mean and covariance. Without corr, an MNN trained with mean and shuffled covariance (all off-diagonal correlation coefficients set to zero). ANN, an ANN that uses ReLU activation for comparison, trained with mean only.