Scaling and context steer LLMs along the same computational path as the human brain

TL;DR

The paper investigates whether large language models and the human brain process natural language in a shared computational sequence. By aligning MEG responses to word onsets with activations from 22 LLMs across architectures, the study shows that earlier LLM layers align with early brain responses and deeper layers with later responses, a pattern that holds across transformers and non-transformer architectures. Temporal alignment intensifies with model size and context length, follows a logarithmic growth with diminishing returns, and is not driven solely by word predictability. A significant correlation between temporal and alignment scores suggests that brain-like sequential processing emerges under certain conditions, offering insights for brain-inspired AI and the role of memory and context in language comprehension.

Abstract

Recent studies suggest that the representations learned by large language models (LLMs) are partially aligned to those of the human brain. However, whether and why this alignment score arises from a similar sequence of computations remains elusive. In this study, we explore this question by examining temporally-resolved brain signals of participants listening to 10 hours of an audiobook. We study these neural dynamics jointly with a benchmark encompassing 22 LLMs varying in size and architecture type. Our analyses confirm that LLMs and the brain generate representations in a similar order: specifically, activations in the initial layers of LLMs tend to best align with early brain responses, while the deeper layers of LLMs tend to best align with later brain responses. This brain-LLM alignment is consistent across transformers and recurrent architectures. However, its emergence depends on both model size and context length. Overall, this study sheds light on the sequential nature of computations and the factors underlying the partial convergence between biological and artificial neural networks.

Figures (15)

• Figure 1: Methods. A. Subjects listened to 10 hours of audio books in the MEG scanner. B. The same text is input to an LLM, e.g. Llama 3-8B. Colors indicate layer depth. To compare this set of - biological and artificial - neural embeddings, we fit a linear mapping W for each layer, and evaluate its accuracy with a Pearson correlation metric: the alignment score $R_{\text{layer}}$. C. Alignment score ($R_{\text{layer}}$) of 9 representative layers of Llama 3-8B, as a function of word-onset (t=0). D. The timestep of peaking alignment scores ($T_{\text{max}}$, x-axis) is plotted for each layer (y-axis). The resulting Temporal score $r$ and associated $p$ are printed on the plot.
• Figure 2: Human brain and LLMs exhibit temporal alignment. Correlation between time of peaking alignment scores ($T_{\text{max}}$, x-axis) and layer depth shows a highly significant temporal alignment. A. Alignment scores of 9 representative layers across each of the 9 studied LLMs, as a function of word-onset (t=0). Alignment scores have been averaged across models. In dashed gray curves, layers from unpretrained versions of these models, averaged over models. B. The time steps of peaking alignment scores ($T_{\text{max}}$, x-axis) are plotted for each representative layer (y-axis), averaged across models. The Temporal score $r$ and associated $p$ are printed on the figure. The grey area indicates the confidence intervals of the regression estimate. Error bars across subjects could not be computed due to the low number of subjects and the need to average across subjects to denoise neural data, though we ensure reproducibility of results across subjects in App. \ref{['app:main_fig_across_subjects']},\ref{['app:alignment_scores_ci_across_subjects']}. Here, colored error bars indicate standard deviations of the layer-wise distributions of $T_{\text{max}}$ across the 9 presented models. C. Temporal scores are computed and presented for each model studied independently. An asterix next to the Temporal score indicates the score is significant with $p$ < 5e-3. D. Alignment scores of 9 representative layers across each of the 9 presented LLMs, as a function of word-onset (t=0). Each figure presents one model studied independently.
• Figure 3: Temporal alignment emerges with model size. Colors indicate layer depth. A. Each of the 5 figures on the horizontal axis presents results for a specific model belonging to the Pythia family and studied independently, of size 14m, 160m, 1b, 6.9b and 12b parameters respectively (left to right). The Pythia family hosts 8 models of increasing size, all trained with the same data amount and parameter choices. Figures present evolution of alignment scores $R_{\text{layer}}$ of 9 representative layers, from 10% to 90% of model depth, as a function of word-onset (t=0). B. Each of the 5 figures on the horizontal axis presents results for a specific models belonging to the Pythia family and studied independently. The time steps of peaking alignment scores ($T_{\text{max}}$, x-axis) are plotted for each representative layer (y-axis). The Temporal score $r$ and associated $p$ are printed on the figure. The grey area indicates the confidence intervals of the regression estimate. C. Temporal and alignment scores as functions of model size, for the 8 models forming the Pythia family. The model names (x-axis) are displayed on a logarithmic scale corresponding to their respective size. The Pearson scores $r$ and associated $p$ quantifying these correlations are printed on the figure. The grey and blue areas indicate the confidence intervals of the regression estimates.
• Figure 4: Temporal alignment increases with the length of the context provided to the LLM. Colors indicate layer depth. A. Each of the four figures on the horizontal axis presents results for a specific context length provided to Llama-3.2 3B, 1-word, 10-word, 500-word and 1000-word contexts respectively (from left to right). Figures present evolution of alignment scores $R_{\text{layer}}$ of 9 representative layers, from 10% to 90% of model depth, as a function of word-onset (t=0). B. Each of the four figures on the horizontal axis presents results for a specific context length provided to Llama-3.2 3B. The time steps of peaking alignment scores ($T_{\text{max}}$, x-axis) are plotted for each representative layer (y-axis). The Temporal score $r$ and associated $p$ are printed on the figure. The grey area indicates the confidence intervals of the regression estimate. C. Temporal and alignment scores as functions of context length when given to Llama-3.2 3B, for six context lengths (x-axis). Context lengths are displayed on a logarithmic scale. The Pearson scores $r$ and associated $p$ quantifying these correlations are printed on the figure. The grey and blue areas indicate the confidence intervals of the regression estimates.
• Figure 5: Temporal alignment holds independently of word predictability. Colors indicate layer depth. A. Alignment scores of 9 representative layers of Llama-3-8B, from 10% to 90% of layer depth, as a function of word-onset (t=0). Alignment score dynamic curves resulting from evaluating only the quartile of most expected words (from context) among the 270 000 words forming the dataset. The contextual predictability of words is computed through Llama-3-8B. B. The time steps of peaking alignment scores ($T_{\text{max}}$, x-axis) of the quartile of most expected words are plotted for each representative layer (y-axis) of Llama-3-8B. The Temporal score $r$ and associated $p$ are printed on the figure. The grey area indicates the confidence intervals of the regression estimate. C. Alignment scores of 9 representative layers of Llama-3-8B, from 10% to 90% of layer depth, as a function of word-onset (t=0). Alignment score dynamic curves resulting from evaluating only the quartile of least expected (i.e. more surprising) words from context, among the 270 000 words forming the dataset. The contextual predictability of words is computed through Llama-3-8B. D. The time steps of peaking alignment scores ($T_{\text{max}}$, x-axis) of the quartile of least expected words are plotted for each representative layer (y-axis) of Llama-3-8B. The Temporal score $r$ and associated $p$ are printed on the figure. The grey area indicates the confidence intervals of the regression estimate. E. The pairwise differences between time steps of peaking alignment scores (Difference in $T_{\text{max}}$, x-axis) per layer, between the quartile of most expected words and the quartile of least expected words, for each representative layer (y-axis) of Llama-3-8B. The Pearson score $r$ and associated $p$ quantifying this correlation are printed on the figure. The grey area indicates the confidence intervals of the regression estimate.