Linear Readout of Neural Manifolds with Continuous Variables

TL;DR

A statistical-mechanical theory of regression capacity that relates linear decoding efficiency of continuous variables to geometric properties of neural manifolds and applies to real data, revealing increasing capacity for decoding object position and size along the monkey visual stream.

Abstract

Brains and artificial neural networks compute with continuous variables such as object position or stimulus orientation. However, the complex variability in neural responses makes it difficult to link internal representational structure to task performance. We develop a statistical-mechanical theory of regression capacity that relates linear decoding efficiency of continuous variables to geometric properties of neural manifolds. Our theory handles complex neural variability and applies to real data, revealing increasing capacity for decoding object position and size along the monkey visual stream.

Figures (7)

• Figure 1: a, Neural manifolds arising from neural responses to images with orientation $\theta$ in $P=3$ disjoint stimulus intervals, recorded from $N=3$ neurons. b, The efficiency of manifold organization is assessed by testing if labels can be linearly read out. Left: manifolds are $\varepsilon$-regressible if they lie within $-\varepsilon \leq \mathbf{w} \cdot \mathbf{x} - y^\mu \leq \varepsilon$, where $\mathbf{w}$ is the regression vector. $\mathbf{w}'$ is an example of non-regressor vector and $\cdot$ stands for inner product. Right: Visualizing the solution space of regressor vectors.
• Figure 2: a, In mean-field theory, we randomly generate $P$ manifolds in $\mathbb{R}^N$ (with $P,N\to\infty$), and define capacity as the maximum load $P/N$ while the Gardner volume (dark green area) is not zero. b, In instance-based theory, $P, N$ are finite and fixed (e.g., given by the data), and we consider randomly projecting the manifolds from $\mathbb{R}^N$ to $\mathbb{R}^{N_\mathrm{proj}}$, define critical dimension $N_\textsf{crit}$ as the smallest $N_\mathrm{proj}$ with at least 0.5 probability of being linearly regressible (yellow star), and define capacity as $\alpha=P/N_\textsf{crit}$.
• Figure 3: Mean-field point-like manifolds. a, $r$ controls the average norm of the points $\{\mathbf{u}_0^\mu\}$, $\sigma$ controls the scale of the target labels $\{y^\mu\}$, and $\psi,\rho \in [0,1)$ set the correlation strengths of the data points and labels respectively. b, Varying one of $\varepsilon, \sigma, r, \psi, \rho$ (others fixed), we numerically estimating capacity with $P=500$ manifolds (see SM SM). The black line shows the theoretical capacity from Equation Eq. \ref{['eq:uncorr-points-cap']} in the $P \to \infty$ limit, where capacity depends monotonically only on the asymptotically equivalent tolerance$\varepsilon_\mathrm{equiv} = \varepsilon/(\sigma\sqrt{1-\rho})$.
• Figure 4: Mean-field sphere-like manifolds. a, Manifold centers and labels are drawn from the correlated points model in \ref{['fig:points']}. $R$ sets the mean radius; $\gamma\in [0,1)$ controls axis correlations. b, Theory–numeric check. Solid lines show the analytic formula Eq. \ref{['eq:corrspherecap']}, and dots indicate numerical estimates with $P=250$ manifolds (see Section VII). Each curve corresponds to a different $D$, with capacity decreasing in $D$. Top: capacity depends on $\varepsilon, \sigma, \rho$ only through the asymptotically equivalent tolerance$\varepsilon_\mathrm{equiv} = \frac{\varepsilon}{\sigma\sqrt{1-\rho}}$ and increases with $\varepsilon_\mathrm{equiv}$. Bottom: capacity depends on $R, r, \psi, \gamma$ only through the asymptotically equivalent radius$R_\mathrm{equiv} = \frac{R\sqrt{1-\gamma}}{r\sqrt{1-\psi}}$ and decreases with $R_\mathrm{equiv}$.
• Figure 5: Application to neuroscience data. a, We analyze electrophysiological recordings from macaque ventral stream as monkeys viewed objects with variable pose parameters (e.g., size, position) superimposed on complex backgrounds. b, For each target parameter (e.g., size), we evenly bin its range into 5 intervals, yielding $P = 5$ manifolds. Each manifold consists of neural responses to 15 individual stimuli whose non-target pose parameters can vary freely, contributing to neural response variability. The background also varies within each manifold and provides another source of nuisance variability. We observe that capacity for each variable increases (i.e., $N_\mathrm{crit}$ decreases) from pixels to V4 to IT, indicating more efficient representations at higher processing stages. Our analytical estimates (solid lines) match numerical results (dots).

Theorems & Definitions (18)

• Definition 1: Simulation capacity
• Definition 2: Capacity formula
• Theorem 1: Formulas for critical dimension and instance-based capacity
• Definition 3
• Lemma 1: Approximate conic kinematic formula, Amelunxen2014
• proof : Proof of Theorem \ref{['ddcap']}.
• Lemma 2
• proof : Proof of \ref{['lem:prob cone']}
• Theorem 2: Formula for fixed-label mean-field manifolds
• proof : Proof of \ref{['corrcap']}