TopoLM: brain-like spatio-functional organization in a topographic language model

TL;DR

TopoLM is developed, a transformer language model with an explicit two-dimensional spatial representation of model units that predicts the emergence of the spatio-functional organization of a cortical language system as well as the organization of functional clusters selective for fine-grained linguistic features empirically observed in human cortex.

Abstract

Neurons in the brain are spatially organized such that neighbors on tissue often exhibit similar response profiles. In the human language system, experimental studies have observed clusters for syntactic and semantic categories, but the mechanisms underlying this functional organization remain unclear. Here, building on work from the vision literature, we develop TopoLM, a transformer language model with an explicit two-dimensional spatial representation of model units. By combining a next-token prediction objective with a spatial smoothness loss, representations in this model assemble into clusters that correspond to semantically interpretable groupings of text and closely match the functional organization in the brain's language system. TopoLM successfully predicts the emergence of the spatio-functional organization of a cortical language system as well as the organization of functional clusters selective for fine-grained linguistic features empirically observed in human cortex. Our results suggest that the functional organization of the human language system is driven by a unified spatial objective, and provide a functionally and spatially aligned model of language processing in the brain.

Figures (14)

• Figure 1: Building a topographic language model with brain-like spatio-functional organization.(a) TopoLM modifies the Transformer architecture with a two-dimensional spatial encoding at the output of each attention and MLP layer. This representation enables the use of a spatial correlation loss that encourages smooth response profiles in adjacent units. This spatial loss is jointly optimized with cross-entropy task loss during training. (b) At each forward pass, we randomly select five neighborhoods in each layer (of which we here only show 3 for clearity) and compute the pairwise correlation of unit activations within each neighborhood. The spatial loss is computed by comparing these correlations to the inverse distances between associated unit pairs, with the final loss averaged across units pairs and neighborhoods. Computing the loss on cortical neighborhoods is an efficient approximation of the spatial loss. (c) We use a FWHM filter to simulate the fMRI sampling process such that a simulated voxel's response ('fMRI-like signal') is composed of a combination of responses from neighboring units kriegeskorte_how_2010. We simulate the response as a Gaussian random variable, with FWHM $2.0$ mm, assuming unit distances of $1.0$ mm.
• Figure 2: Brain-like response profiles across the core language system.(a) Applying a functional localizer fedorenko_new_2010 we isolate the core language system of TopoLM, and find clear brain-like spatial organization (for brevity, we only show Transformer blocks 5-12 here). Response profile across individual language-selective clusters (shown in yellow) in TopoLM are similar to one another, consistent with (b) the language system in human cortex fedorenko_language_2024. (c) Across the entire core language system, TopoLM (blue) mostly matches the neural data (green), but not exactly; however, the non-topographic baseline model (orange) fails to capture neural patterns as well.
• Figure 3: Brain-like verb- and noun-selective clusters in TopoLM.(a) fMRI data from hauptman_neural_2024 points to verb- (red) and noun-selective (blue) regions in the human cortex with strong clustering (Moran's $I = 0.96$). (b) Quantification of clustering. Relative to high clustering in the brain (green dashed line), the non-topographic baseline shows limited clustering (orange). The topographic model shows moderate clustering at the unit level (light blue) and strong clustering when simulating fMRI sampling (dark blue). On stimuli from moseley_nouns_2014 (fMRI data not available) we find qualitatively similar results. (c) Exemplary model maps (last MLP layer) showing the verb-/noun contrast (red-blue) in response to stimuli from hauptman_neural_2024. The non-topographic baseline shows no clustering while the topographic model develops verb- and noun-selective clusters.
• Figure 4: Verb- and noun-selectivity in response to concrete and abstract stimuli.(a) Using stimuli from moseley_nouns_2014, we find verb-/noun-selective clusters (verb: red / noun: blue) emerging in TopoLM for concrete (solid lines), but not for abstract words (dashed lines), thus replicating their results. (b) We obtain strong verb-/noun-clustering when concrete words are used to compute the verb-/noun-contrast (light blue, solid lines), $I = 0.80$), but substantially lower clustering for abstract words ($I = 0.23$, light blue, dashed lines, $t$-test: $p < 0.001$). However, we do not find evidence for such a difference in verb-/noun-clustering when using the non-topographic control ($I = 0.11$ vs. $0.12$, $t$-test: $p>0.05$, orange). Results do not change qualitatively when fMRI readout sampling is performed before computing contrasts and clustering (dark blue). In all the presented cases, spatial autocorrelation is computed on un-thresholded maps (for all layers, see Figure \ref{['fig:moseley_model_maps_appendix']}). Following neuroimaging conventions on defining category-selective cortical clusters, we show model maps thresholded at $p(\text{FDR}) < 0.05$.
• Figure 5: Brain-Score brain alignment performance (linear predictivity). We tested TopoLM and the non-topographic control on Brain-Score language using linear predictivity to estimate alignment. TopoLM outperforms the control in Blank2014 and Tuckete2024, but performs slightly worse at Pereira2018 and Fedorenko2016 ($t$-test performed for each benchmark separately: $p<0.05$). For each benchmark, we sampled from 10 cross-validation loops to compute the bootstrapped 95% confidence intervals (black bars). When results are averaged across the 4 benchmarks (last panel 'Brain-Score'), we don't find evidence for a significant difference between TopoLM and its control ($t$-test: $p>0.05$).