Simple Models, Rich Representations: Visual Decoding from Primate Intracortical Neural Signals

TL;DR

This paper tackles visual decoding from high-density intracortical neural signals in primates. It shows that modeling temporal dynamics with a lightweight temporal-attention module feeding a shallow MLP to semantic embeddings suffices for high-quality decoding, outperforming many deeper architectures. A two-stage, modular pipeline combines retrieval in CLIP embedding space with open-ended image generation via diffusion models, achieving plausible reconstructions from about 200 ms of brain activity. The results provide design principles for brain–computer interfaces and semantic neural decoding, including practical scaling laws for data and channel dimensionality.

Abstract

Understanding how neural activity gives rise to perception is a central challenge in neuroscience. We address the problem of decoding visual information from high-density intracortical recordings in primates, using the THINGS Ventral Stream Spiking Dataset. We systematically evaluate the effects of model architecture, training objectives, and data scaling on decoding performance. Results show that decoding accuracy is mainly driven by modeling temporal dynamics in neural signals, rather than architectural complexity. A simple model combining temporal attention with a shallow MLP achieves up to 70% top-1 image retrieval accuracy, outperforming linear baselines as well as recurrent and convolutional approaches. Scaling analyses reveal predictable diminishing returns with increasing input dimensionality and dataset size. Building on these findings, we design a modular generative decoding pipeline that combines low-resolution latent reconstruction with semantically conditioned diffusion, generating plausible images from 200 ms of brain activity. This framework provides principles for brain-computer interfaces and semantic neural decoding.

Figures (7)

• Figure 1: Overview of our generative decoding framework. Neural activity is recorded from a macaque implanted with a 1024-channel MUA array while viewing 22,000 natural images. A two-branch decoding pipeline is used: (top) a structural decoder maps neural signals to a low-resolution latent representation using a VAE decoder, providing a faithful but coarse reconstruction; (bottom) a semantic decoder generates multiple candidate images via Stable Diffusion XL conditioned on semantic information decoded from the same neural signal. The final image is selected by computing the SSIM between each candidate and the structural image, effectively combining the structural accuracy of the low-res decoder with the generative power of the semantic branch.
• Figure 2: Overview of the proposed neural-to-semantic visual decoding pipeline. Multielectrode MUA responses evoked by visual stimuli are preprocessed and fed into a temporal-attention module that learns stimulus-dependent weighting over time, followed by a shallow MLP that maps neural activity into the CLIP embedding space. Training is supervised through a contrastive loss that aligns predicted neural embeddings with ground-truth CLIP representations of the viewed images. At test time, the predicted embeddings support two complementary tasks: (i) retrieval, where the model selects the most semantically similar image from a candidate set, and (ii) generation, where the embedding conditions a Stable Diffusion model (via an IP-Adapter) to synthesize novel images preserving the semantics inferred from neural activity. A rejection-sampling strategy ensures that generated samples remain structurally consistent with the original ones.
• Figure 3: Left: Top-1 and Top-5 classification accuracy as a function of PCA dimensionality applied to neural channels. Accuracy increases logarithmically with dimensionality, as shown by high R$^2$ in the log-fit curves. Right: Top-1 and Top-5 accuracy as a function of training set size. Performance scales log-linearly with data, underscoring the importance of dataset size in brain-based visual decoding.
• Figure 4: Heatmap of attention weights for the test set (right side) and average weights plot over the entire set (left side). Warmer colors of the heatmap indicate higher attention weights.
• Figure 5: Examples of image reconstructions from neural activity. Each triplet shows the original image (left), a low-resolution baseline (middle), and our reconstruction (right) with corresponding SSIM score. The generations capture essential attributes such as object structure, color distribution, and general content.