MEIcoder: Decoding Visual Stimuli from Neural Activity by Leveraging Most Exciting Inputs

TL;DR

MEIcoder addresses the challenge of decoding high-fidelity visual stimuli from scarce neural data by introducing neuron-specific Most Exciting Inputs (MEIs) as a biologically grounded prior, coupled with a structural similarity (SSIM) loss and adversarial training. It uses a shared core CNN across subjects with per-subject readins, enabling data- and neuron-efficient learning that scales to 1,000–2,500 neurons with limited training data. Across Brainreader, SENSORIUM 2022, and synthetic Cat V1 datasets, MEIcoder achieves state-of-the-art reconstructions and provides a unified benchmark with over 160k samples. The work offers interpretability analyses and scaling insights, advancing practical neural decoding for neuroscience and neuroengineering while highlighting limitations and considerations for generalization to higher visual areas and humans.

Abstract

Decoding visual stimuli from neural population activity is crucial for understanding the brain and for applications in brain-machine interfaces. However, such biological data is often scarce, particularly in primates or humans, where high-throughput recording techniques, such as two-photon imaging, remain challenging or impossible to apply. This, in turn, poses a challenge for deep learning decoding techniques. To overcome this, we introduce MEIcoder, a biologically informed decoding method that leverages neuron-specific most exciting inputs (MEIs), a structural similarity index measure loss, and adversarial training. MEIcoder achieves state-of-the-art performance in reconstructing visual stimuli from single-cell activity in primary visual cortex (V1), especially excelling on small datasets with fewer recorded neurons. Using ablation studies, we demonstrate that MEIs are the main drivers of the performance, and in scaling experiments, we show that MEIcoder can reconstruct high-fidelity natural-looking images from as few as 1,000-2,500 neurons and less than 1,000 training data points. We also propose a unified benchmark with over 160,000 samples to foster future research. Our results demonstrate the feasibility of reliable decoding in early visual system and provide practical insights for neuroscience and neuroengineering applications.

Figures (14)

• Figure 1: Architecture of MEIcoder. First, individual neural responses $\mathbf{r}_i$ and their corresponding neuron embeddings $\mathbf{e}_i$ get projected by a feed-forward network into context representations $\mathbf{C}_i$ (left). The context representations are then pointwise-multiplied with MEIs and passed through a pointwise convolution layer (center). Finally, the output of the convolution layer is used by the CNN core to reconstruct the final image (right).
• Figure 2: Reconstructions on the Brainreader (top) and SENSORIUM 2022 (bottom) datasets.
• Figure 3: Synthetic Cat V1 reconstructions. See \ref{['sec:appendix:reconstructions']} for more examples.
• Figure 4: Ablation study on sub-components of MEIcoder. The percentage (y-axis) is calculated with respect to a setting with no ablations. Reported values are the average across the three datasets.
• Figure 5: Variance in reconstruction quality.