Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adaptive whitening in neural populations with gain-modulating interneurons

Lyndon R. Duong, David Lipshutz, David J. Heeger, Dmitri B. Chklovskii, Eero P. Simoncelli

TL;DR

This work addresses adaptive whitening in neural populations by proposing a biologically plausible, gain-modulated mechanism that operates online with fixed synaptic weights. It introduces a novel objective that replaces the traditional $\mathbf{C}_{yy}=\mathbf{I}_N$ constraint with marginal-variance constraints across an overcomplete frame $\mathbf{W}$, encoded by adaptive gains $\mathbf{g}$ via the factorization $\mathbf{C}_{xx}^{1/2}=\mathbf{W}\operatorname{diag}(\mathbf{g})\mathbf{W}^T+\mathbf{I}_N$. An online RNN is derived, featuring fast neural dynamics that converge to $\bar{\mathbf{y}}_t=[\mathbf{I}_N+\mathbf{W}\operatorname{diag}(\mathbf{g})\mathbf{W}^T]^{-1}\mathbf{x}_t$ and slow gain updates $\mathbf{g}\leftarrow\mathbf{g}+\eta(\bar{\mathbf{z}}_t^{\circ 2}-\mathbf{1})$, enabling unsupervised, run-time whitening without backpropagation. Numerical experiments show that sign-constrained gains enhance robustness to ill-conditioned inputs, that the convergence rate depends on the frame $\mathbf{W}$, and that local spatial whitening can be achieved with convolutional frames, offering a practical path to online, low-power decorrelation in biological and machine learning systems.

Abstract

Statistical whitening transformations play a fundamental role in many computational systems, and may also play an important role in biological sensory systems. Existing neural circuit models of adaptive whitening operate by modifying synaptic interactions; however, such modifications would seem both too slow and insufficiently reversible. Motivated by the extensive neuroscience literature on gain modulation, we propose an alternative model that adaptively whitens its responses by modulating the gains of individual neurons. Starting from a novel whitening objective, we derive an online algorithm that whitens its outputs by adjusting the marginal variances of an overcomplete set of projections. We map the algorithm onto a recurrent neural network with fixed synaptic weights and gain-modulating interneurons. We demonstrate numerically that sign-constraining the gains improves robustness of the network to ill-conditioned inputs, and a generalization of the circuit achieves a form of local whitening in convolutional populations, such as those found throughout the visual or auditory systems.

Adaptive whitening in neural populations with gain-modulating interneurons

TL;DR

This work addresses adaptive whitening in neural populations by proposing a biologically plausible, gain-modulated mechanism that operates online with fixed synaptic weights. It introduces a novel objective that replaces the traditional constraint with marginal-variance constraints across an overcomplete frame , encoded by adaptive gains via the factorization . An online RNN is derived, featuring fast neural dynamics that converge to and slow gain updates , enabling unsupervised, run-time whitening without backpropagation. Numerical experiments show that sign-constrained gains enhance robustness to ill-conditioned inputs, that the convergence rate depends on the frame , and that local spatial whitening can be achieved with convolutional frames, offering a practical path to online, low-power decorrelation in biological and machine learning systems.

Abstract

Statistical whitening transformations play a fundamental role in many computational systems, and may also play an important role in biological sensory systems. Existing neural circuit models of adaptive whitening operate by modifying synaptic interactions; however, such modifications would seem both too slow and insufficiently reversible. Motivated by the extensive neuroscience literature on gain modulation, we propose an alternative model that adaptively whitens its responses by modulating the gains of individual neurons. Starting from a novel whitening objective, we derive an online algorithm that whitens its outputs by adjusting the marginal variances of an overcomplete set of projections. We map the algorithm onto a recurrent neural network with fixed synaptic weights and gain-modulating interneurons. We demonstrate numerically that sign-constraining the gains improves robustness of the network to ill-conditioned inputs, and a generalization of the circuit achieves a form of local whitening in convolutional populations, such as those found throughout the visual or auditory systems.
Paper Structure (42 sections, 3 theorems, 49 equations, 10 figures, 3 algorithms)

This paper contains 42 sections, 3 theorems, 49 equations, 10 figures, 3 algorithms.

Table of Contents

  1. Introduction
  2. A Novel Objective for Symmetric Whitening
  3. Conventional objective
  4. A novel objective using marginal statistics
  5. An RNN with Gain Modulation for Adaptive Symmetric Whitening
  6. Enforcing the marginal variance constraints with scalar gains
  7. Deriving RNN neural dynamics and gain updates
  8. Fast neural activity dynamics
  9. Slow gain dynamics
  10. Online unsupervised algorithm
  11. Numerical Experiments and Applications
  12. Adaptive symmetric whitening via gain modulation
  13. Algorithmic convergence rate depends on ${\mathbf{W}}$
  14. Implicit sparse gating via gain modulation
  15. Normalizing ill-conditioned data
  16. ...and 27 more sections

Key Result

Proposition 2.1

Fix $K\geq K_N$. Suppose ${\mathbf{w}}_1,\dots,{\mathbf{w}}_K\in\mathbb{R}^N$ are unit vectorsThe unit-length assumption is imposed, without loss of generality, for notational convenience. such that where ${\mathbb{S}}^N$ denotes the $K_N$-dimensional vector space of $N\times N$ symmetric matrices. Then eq:Cyy holds if and only if the projection of ${\mathbf{y}}_t$ onto each unit vector ${\mathbf

Figures (10)

  • Figure 1: Schematic of a recurrent statistical whitening network with 2 primary neurons and 3 interneurons. Left: 2D Scatter plot of network inputs ${\mathbf{x}}=[x_1,x_2]^\top$ (e.g. post-synaptic currents), with covariance indicated by the ellipse. Center: Primary neurons, with outputs ${\mathbf{y}}=[y_1, y_2]^\top$, receive external feedforward inputs, ${\mathbf{x}}$, and recurrent feedback from an overcomplete population of interneurons, $-\sum_{i=1}^3g_iz_i{\mathbf{w}}_i$. Projection vectors $\{{\color{RubineRed}{\mathbf{w}}_1}, {\color{ForestGreen}{\mathbf{w}}_2}, {\color{Cerulean}{\mathbf{w}}_3}\}\in\mathbb{R}^2$ encode feedforward synaptic weights connecting primary neurons to interneuron $i=1,2,3$, with symmetric feedback connections. Weight vectors are shown in the left and right panels with corresponding colors. In general, the network may require all-to-all connectivity between primary and interneurons; we use a reduced subset of connections here for diagram clarity. Inset: The $i$th interneuron (e.g. here $i=2$) receives input $z_i={\mathbf{w}}_i^\top{\mathbf{y}}$, which is multiplied by its gain $g_i$ to produce output $g_iz_i$. Its gain, $g_i$, is adjusted s.t. $\Delta g_i \propto z_i^2-1$. The dark arrow indicates that the gain update operates on a slower time scale. Right: Scatter plots of the whitened network outputs ${\mathbf{y}}$. Outputs have unit variance along all ${\mathbf{w}}_i$'s, which is equivalent to having identity covariance matrix, i.e., ${\mathbf{C}}_{yy}={\mathbf{I}}_N$ (black circle).
  • Figure 2: Network from \ref{['fig:schematique']} (with corresponding colors; $N{=}2$, $K{=}K_N{=}3$, $\eta{=}$2E-3) adaptively whitening samples from two randomly generated statistical contexts online (10K steps each). Top: Marginal variances measured by interneurons approach 1 over time. Middle: Dynamics of interneuron gains, which are applied to $z_i$ before feeding back onto the primary cells. Dashed lines are optimal gains (\ref{['appendix:psd_frame']}). Bottom: Error over time, as measured by the maximal difference between the standard deviation along the principal axes of ${\mathbf{C}}_{yy}$ and unity.
  • Figure 3: Convergence rate depends on structure of ${\mathbf{W}}$. For each network, $\eta{=}$1E-2. A: Error over time. Curves are median and [25%, 75%] quantile regions over 100 repeats. Dashed line indicates when the principal axes of 1-standard deviation ellipse representing ${\mathbf{C}}_{yy}$ are within 0.1 of unity. B: Scatter plots and covariance ellipses of ${\mathbf{y}}$ for a single experiment with each frame type at different steps. Gray dashed lines are axes spanned by ${\mathbf{W}}$.
  • Figure 4: Gain modulation as a fast implicit sparse gating mechanism. Top: Error over time for Spectral vs. Random networks ($N{=}6$; $K{=}K_N{=}21$; $\eta{=}$1E-3) adapting to 2 alternating statistical contexts with different input covariances. Dashed line indicates when the principal axes of 1-standard deviation ellipsoid representing ${\mathbf{C}}_{yy}$ are within 0.1 of unity. Bottom: Gains act as implicit context switches, sparsely gating the respective eigenbases embedded in the Spectral frame to optimally whiten each context.
  • Figure 5: Two networks ($N{=}2$, $K{=}3$, $\eta{=}0.02$) whitening ill-conditioned inputs. A: Outputs without whitening. 2D scatterplot of a non-Gaussian density whose underlying signal lies close to a latent 1D axis. Many points lie outside of the axis limits in this panel. Signal magnitude along that axis is denoted by the grayscale gradient. The 1-standard deviation covariance matrix is depicted as a black ellipse. Colored lines are axes spanned by Optimal frame (see Sec \ref{['ssec:convergence']}). B: Symmetric whitening boosts noise along the uninformative direction. C: Modulating gains according to Eq. \ref{['eq:grectified']} rescales the data without amplifying noise. D: Gains updated with Eq. \ref{['eq:gupdate']} vs. Eq. \ref{['eq:grectified']}. Colors correspond to frame axes in panels A--C.
  • ...and 5 more figures

Theorems & Definitions (4)

  • Proposition 2.1
  • Lemma 2.1: Von Neumann’s trace inequality
  • proof : Proof of Proposition \ref{['prop:marginal']}
  • Theorem 5.1