Convex Efficient Coding
William Dorrell, Peter E. Latham, James Whittington
TL;DR
This work develops a tractable, convex framework for normative neural coding by reformulating optimizations over neural representations as convex problems on the representational similarity matrix $\mathbf{Q}=\mathbf{Z}^T\mathbf{Z}$. By showing convexity across a wide suite of constraints and objectives, the authors construct a family of problems that encompass linear and certain nonlinear networks, including nonnegative affine autoencoders and nonlinear similarity matching. They derive a tight identifiability criterion for semi-nonnegative matrix factorisation, establish when population-level representational similarity uniquely determines single-neuron tunings, and use tractable nonlinear formulations to explain ON-OFF coding in retinal-like settings, with sparsity setting the transition between single-channel and dual-channel codes. The framework thus links population representations to neuron-level tuning, enabling analytic insights into neural codes and potentially guiding interpretability in AI models. Overall, the paper provides a versatile, convex analytic toolkit for addressing core questions in neural coding and representation learning, with implications for both neuroscience and machine learning interpretability.
Abstract
Why do neurons encode information the way they do? Normative answers to this question model neural activity as the solution to an optimisation problem; for example, the celebrated efficient coding hypothesis frames neural activity as the optimal encoding of information under efficiency constraints. Successful normative theories have varied dramatically in complexity, from simple linear models (Atick & Redlich '90), to complex deep neural networks (Lindsay '21). What complex models gain in flexibility, they lose in tractability and often understandability. Here, we split the difference by constructing a set of tractable but flexible normative representational theories. Instead of optimising the neural activities directly, following Sengupta et al. '18, we optimise the representational similarity, a matrix formed from the dot products of each pair of neural responses. Using this, we show that a large family of interesting optimisation problems are convex. This family includes problems corresponding to linear and some non-linear neural networks, and problems from the literature not previously recognised as convex, such as modified versions of semi-nonnegative matrix factorisation or nonnegative sparse coding. We put these findings to work in three ways. First, we provide the first necessary and sufficient identifiability result for a form of semi-nonnegative matrix factorisation. Second, we show that if neural tunings are `different enough' then they are uniquely linked to the optimal representational similarity, partially justifying the use of single neuron tuning analysis in neuroscience. Finally, we use the tractable nonlinearity of some of our problems to explain why dense retinal codes, but not sparse cortical codes, optimally split the coding of a single variable into ON & OFF channels. In sum, we identify a space of convex problems, and use them to derive neural coding results.