How 'Neural' is a Neural Foundation Model?

TL;DR

This work interrogates a leading neural foundation model (FNN) by peering inside its units with neuroscience-inspired manifolds to assess how brain-like its representations are. It combines decoding manifolds (stimulus-space structure), encoding manifolds (neuron-space topology), and decoding trajectories (dynamic population activity), using metrics such as RSA, CCA, LP, DSA, and novel tubularity scores to quantify alignment with mouse retina/V1 data. The recurrent module emerges as the primary source of brain-like temporal structure, while the encoder and readout show divergences from biology that suggest concrete architectural refinements, such as earlier recurrence and more diverse, biologically plausible readout features. Overall, the study demonstrates that while FNNs can mimic certain neural dynamics, achieving closer brain- alignment requires targeted architectural constraints and a richer representation of temporal processing, guiding future design of brain-aligned foundation models.

Abstract

Foundation models have shown remarkable success in fitting biological visual systems; however, their black-box nature inherently limits their utility for understanding brain function. Here, we peek inside a SOTA foundation model of neural activity (Wang et al., 2025) as a physiologist might, characterizing each 'neuron' based on its temporal response properties to parametric stimuli. We analyze how different stimuli are represented in neural activity space by building decoding manifolds, and we analyze how different neurons are represented in stimulus-response space by building neural encoding manifolds. We find that the different processing stages of the model (i.e., the feedforward encoder, recurrent, and readout modules) each exhibit qualitatively different representational structures in these manifolds. The recurrent module shows a jump in capabilities over the encoder module by 'pushing apart' the representations of different temporal stimulus patterns. Our 'tubularity' metric quantifies this stimulus-dependent development of neural activity as biologically plausible. The readout module achieves high fidelity by using numerous specialized feature maps rather than biologically plausible mechanisms. Overall, this study provides a window into the inner workings of a prominent neural foundation model, gaining insights into the biological relevance of its internals through the novel analysis of its neurons' joint temporal response patterns. Our findings suggest design changes that could bring neural foundation models into closer alignment with biological systems: introducing recurrence in early encoder stages, and constraining features in the readout module.

Figures (18)

• Figure 1: Approach and manifolds analysisA Stimulus ensemble provides input. B FNN consists of multiple encoding blocks, modeled as convolutional layers, followed by recurrent and readout/interpolation layers. C The tensor of data, containing the response (in time) of each sampled unit to the stimulus ensemble. D PeriStimulus Time Histogram: The response (instantaneous "firing rate") of a single unit/neuron to a stimulus pattern drifting in each of 8 different directions. The curves are redrawn as an image, with brightness corresponding to activity. A plane through the data tensor shows the PSTHs for each of the 6 stimulus classes, drifting in all directions. E Decoding manifold, plots the total activity for each stimulus in PCA-reduced neural coordinates. Colors correspond to stimulus classes in A. F The time evolution of each stimulus presentation, plotted in PCA-reduced neural coordinates for the early encoder layer. Note the nested, periodic trajectories indicating a stimulus drifting over a receptive field filter. G Encoding manifold plots individual units/neurons in stimulus/response coordinates. Note the clustering of units with similar responses across the ensemble.
• Figure 2: Decoding Manifolds for the mouse (A) retina and (D) visual cortex are highly clustered by stimulus (color labels shown in top-right bar) supporting decoding (i.e., reading out the stimulus from neural responses) in both cases. By contrast, the FNN is most clustered at the recurrent and readout stages (E--H). Acc: classification accuracy for that layer (see Table 1). Notice how the encoder (first stage in the FNN) differs significantly from the retina (first stage in the visual system); on the other hand, the recurrent layer is most analogous to V1.
• Figure 3: Encoding Manifolds for the mouse (A) retina and (D) visual cortex differ significantly: retina is clustered and cortex is continuous. Example PSTHs show how functionality varies smoothly in cortex but not in the retina. (E) The encoder stage showed a distinct arm of orientation-selective units ($\alpha$), which are compatible with biological results, and another of intensity-based units ($\gamma$), which are not. (F) The recurrent stage showed many direction-selective units, but the following (G) readout stage was the most clustered among all stages. This "bottleneck" layer is then interpolated to a continuous (H) output layer. While the topology of this final layer is similar to that of biological visual cortex, the responses of individual units (PSTHs) are not.
• Figure 4: Decoding Trajectories in the retina (A) and V1 (D) show the development of neural activity dynamics into stimulus tubes. The encoder (E) shows only a non-selective increase in activity (see also Figure \ref{['fig:no_intensity']}) rather than stimulus-dependent tubes. From the recurrent stage onward (F--H), tubular trajectories similar to those seen in biological data are present. The tubularity metrics quantify this phenomenon ($S_{tight}$), and also highlight a lack of complexity in FNN activity compared to the biological data, reflected in their lower crossings values ($S_{cross}$).
• Figure 5: FNN architecture. Layers used for sampling are highlighted. Modulation module omitted as it has no effect for our analysis. The FNN used GeLU activations in the convulutional layers, and Tanh activations in the Recurrent module.