Deep Models, Shallow Alignment: Uncovering the Granularity Mismatch in Neural Decoding

TL;DR

This work identifies a fundamental granularity mismatch in neural visual decoding: neural signals contain multi-scale texture and semantic information, while final-layer vision embeddings suppress low-level details. It introduces Shallow Alignment, which targets intermediate vision representations and uses a linear projector with a symmetric contrastive loss to align neural signals with these representations. The proposed approach yields substantial gains across THINGS-EEG and THINGS-MEG benchmarks and reveals that larger vision backbones can better decode neural data when aligned at appropriately chosen intermediate layers, effectively unlocking scaling laws. The findings imply a closer alignment between human visual processing and hierarchical deep representations and offer a practical pathway to improve non-invasive neural decoding for brain-computer interfaces.

Abstract

Neural visual decoding is a central problem in brain computer interface research, aiming to reconstruct human visual perception and to elucidate the structure of neural representations. However, existing approaches overlook a fundamental granularity mismatch between human and machine vision, where deep vision models emphasize semantic invariance by suppressing local texture information, whereas neural signals preserve an intricate mixture of low-level visual attributes and high-level semantic content. To address this mismatch, we propose Shallow Alignment, a novel contrastive learning strategy that aligns neural signals with intermediate representations of visual encoders rather than their final outputs, thereby striking a better balance between low-level texture details and high-level semantic features. Extensive experiments across multiple benchmarks demonstrate that Shallow Alignment significantly outperforms standard final-layer alignment, with performance gains ranging from 22% to 58% across diverse vision backbones. Notably, our approach effectively unlocks the scaling law in neural visual decoding, enabling decoding performance to scale predictably with the capacity of pre-trained vision backbones. We further conduct systematic empirical analyses to shed light on the mechanisms underlying the observed performance gains.

Figures (13)

• Figure 1: Overview of the proposed Shallow Alignment framework. The model aligns neural signals with intermediate visual representations to mitigate information-lossy alignment at high semantic levels.
• Figure 2: Comparative analysis of representational performance between intermediate and final layers on THINGS-EEG. (a) Top 1 accuracies of intermediate features across vision backbones. Relative depth is computed as $(\ell-1)/(L-1)$. Dashed orange lines mark the Final Output performance. For ResNet models, the final feature is obtained by attention pooling of the last convolutional layer. For Transformer-based models, the final feature corresponds to the CLS token embedding from the last layer. (b) Performance comparison across architectures. The bar chart summarizes the Top-1 accuracy gap between the best-performing intermediate layer (orange) and the final output layer (blue). (c) Scaling analysis. Linear regression analysis reveals the relationship between model scale (number of parameters in ln scale) and Top-1 accuracy. Statistical significance is denoted by asterisks (** for $p < 0.01$) or by “ns” for non-significant results ($p > 0.05$).
• Figure 3: Concept accuracy and retrieval Top-1 accuracy on THINGS-EEG versus layer depth for InternViT. Relative depth is computed as $(\ell-1)/(L-1)$.
• Figure 4: Top-5 retrieved samples of Subject 7 on THINGS-EEG. (a) Retrieval based on the best intermediate-layer embeddings. (b) Retrieval based on final output embeddings. The red box indicates the ground-truth target.
• Figure 5: UMAP visualization of cross-modal alignment on THINGS-EEG (Subject 7). We illustrate the feature alignment between EEG signals and visual representations on the test set ($M=200$). Image features are extracted from selected intermediate layers versus the final output layer, aligned with EEG embeddings from the neural encoder. Gray lines connect ground-truth image--EEG pairs.