Manifolds and Modules: How Function Develops in a Neural Foundation Model

TL;DR

The paper probes a state-of-the-art neural foundation model of activity by constructing decoding and encoding manifolds and tracking joint temporal trajectories across encoder, recurrent, and readout modules. By comparing these internal representations to mouse visual cortex data, it finds the recurrent module most strongly supports temporal discrimination, while the encoder shows limited temporal dynamics and the readout introduces rich variability via many feature maps. The authors demonstrate how manifold-based analyses can reveal brain-like structure and where foundation models diverge from biology, suggesting architectural tweaks to improve interpretability and biological plausibility without sacrificing predictive power. This approach advances interpretability of complex foundation models and informs design choices for future neuro-inspired AI systems.

Abstract

Foundation models have shown remarkable success in fitting biological visual systems; however, their black-box nature inherently limits their utility for understanding brain function. Here, we peek inside a SOTA foundation model of neural activity (Wang et al., 2025) as a physiologist might, characterizing each 'neuron' based on its temporal response properties to parametric stimuli. We analyze how different stimuli are represented in neural activity space by building decoding manifolds, and we analyze how different neurons are represented in stimulus-response space by building neural encoding manifolds. We find that the different processing stages of the model (i.e., the feedforward encoder, recurrent, and readout modules) each exhibit qualitatively different representational structures in these manifolds. The recurrent module shows a jump in capabilities over the encoder module by 'pushing apart' the representations of different temporal stimulus patterns; while the readout module achieves biological fidelity by using numerous specialized feature maps rather than biologically plausible mechanisms. Overall, we present this work as a study of the inner workings of a prominent neural foundation model, gaining insights into the biological relevance of its internals through the novel analysis of its neurons' joint temporal response patterns.

Figures (10)

• Figure 1: Set up and techniques.A: The stimuli consist of drifting gratings (at two spatial frequencies and 8 directions of motion) plus dotted and oriented flows, at two contrasts, drifting in 8 directions. B, C: Stimuli exercise both the mouse visual system (data from published literature, image from Wilks13) and the trained FNN (this paper) to yield activity (firing rate) in time for each direction. D: The firing rates are collected as a PeriStimulus Time Histogram (PSTH), denoted as a heatmap image (higher firing rate is brighter). E: Neural decoding manifold (each point is a trial; coordinates are PCA-reduced neural firings); colors for each trial point match the boxes around stimuli in A. While the trials are weakly clustered by stimulus, the representations do not allow for clear classification at this stage. F: Decoding trajectories show development of neural activity over time for each stimulus, also in PCA coordinates. As expected in the early stage feedforward encoder, neural activity barely changes after the onset of activity compared to the 0-activity point (black). Only circular temporal developments are observable for periodic input stimuli, such as moving gratings (light blue). G: Neural encoding manifold, in which each point is a neuron, in diffusion coordinates. Average PSTHs for neurons in circled clusters show average activity for each of the stimulus classes (arranged as in A). Note the multi-selectivity of neurons to different stimulus classes and especially the "amplification" induced by neurons in cluster $\beta$. We study the encoder (shown above), recurrent and readout modules, and ask whether they have analogues in the mouse visual system.
• Figure 2: Encoding and recurrent layers. A: Encoding manifold for final encoding block. PSTHs for arm $\beta$ amplify all stimulus signals; inset shows mean response intensity development $\pm1$ s.e.m. within units in $\beta$ (yellow) compared with others. B: Explosive growth of trajectories for FNN is caused by initial intensity increase in $\beta$. Ensuing temporal dynamics are negligible. This differs from the trajectory bundles found in mouse retina (C), showing stimulus dependence instead of nonselective intensity induced temporal patterns. D: Recurrent hidden state shows multi-selectivity of units and no explosive intensity growth (cf. inset). E: Decoding trajectories show increased stimulus-dependent temporal patterns leading to better discriminability of stimuli in PCA space. However, trajectories are more temporally monotonic than in primary visual cortex (F).
• Figure 3: Tubularity of recurrent module and output compared to mouse data. Similar decoding trajectories can be clustered into bundles and averaged. Here we cluster by stimulus class; the mean contour is displayed in the stimulus-class color for A. the recurrent module (note class separation developing) and B. the output layer (note the 'circular' dynamics). These differ from biological trajectories for C. mouse retina and D. V1. These differences are quantified by the tightness and crossing measures (Appendix \ref{['tubular:methods']}) for both ground truth and unsupervised (HDBSCAN) groupings.
• Figure 4: Contrasting the readout (A) and output (B) layers. While the decoding manifolds and trajectories appear qualitatively similar, the encoding manifolds have remarkably distinct topologies: while the readout module is highly clustered, the output is continuous. The clustered topology is caused by the interpolation step producing a large amount of features with low within-feature variability. The smooth output is obtained by collapsing the many feature maps to a single output by a linear combination.
• Figure 5: Feedforward encoder L8: Supporting information for \ref{['fig2']}. The intensity arm is clearly visible, exhibiting high mean activity. The low temporal variance, low Orientation Selectivity Indices, and unstructured preferred stimuli show the absence of complex activity patterns in this intensity arm.