Modeling the language cortex with form-independent and enriched representations of sentence meaning reveals remarkable semantic abstractness

TL;DR

The paper investigates whether the human language cortex encodes meaning in a highly abstract, form-independent manner by predicting fMRI responses from representations that differ in modality and surface form. It shows that vision-model embeddings from multiple generated images, paraphrase embeddings averaged over many variants, and enriched commonsense paraphrases can predict language cortex activity, often approaching or surpassing original sentence embeddings in predictive power. A key finding is that representational averaging enhances brain predictivity, revealing a shared semantic core across modalities and surface forms. These results imply that the brain maintains richer semantic representations than current language models and highlight the potential of cross-modal and context-enriched perturbations to probe neural meaning processing.

Abstract

The human language system represents both linguistic forms and meanings, but the abstractness of the meaning representations remains debated. Here, we searched for abstract representations of meaning in the language cortex by modeling neural responses to sentences using representations from vision and language models. When we generate images corresponding to sentences and extract vision model embeddings, we find that aggregating across multiple generated images yields increasingly accurate predictions of language cortex responses, sometimes rivaling large language models. Similarly, averaging embeddings across multiple paraphrases of a sentence improves prediction accuracy compared to any single paraphrase. Enriching paraphrases with contextual details that may be implicit (e.g., augmenting "I had a pancake" to include details like "maple syrup") further increases prediction accuracy, even surpassing predictions based on the embedding of the original sentence, suggesting that the language system maintains richer and broader semantic representations than language models. Together, these results demonstrate the existence of highly abstract, form-independent meaning representations within the language cortex.

Figures (28)

• Figure 1: We probe whether neural activity in the language cortex can be explained by different representations of the original linguistic input. Starting from each original sentence, we derive and analyze eight alternative representations - (A) original sentence, (B) content words capturing the main semantic elements, (C) a header phrase summarizing a group of related sentences, (D) paraphrases of the original sentence, (E) paraphrase of the original sentence enriched with commonsense context, (F) images generated from the original sentence, (G) images generated from the content words, and (H) images generated from the header phrase. Language-based variants (A–E) are embedded with large language models, while visual variants (F–H) are embedded using vision models.
• Figure 2: Comparison of performance in predicting language cortex activity between LLM embeddings of the original linguistic stimuli from the Pereira (2018) dataset (CLIP, MPNet, BERT-BASE, GPT2-XL and LLAMA3 presented in grey horizontal lines) and vision model embeddings of their corresponding visual counterparts (remaining models). Performance of the visual models increases with the number of images, sometimes surpassing some of the language models.
• Figure 3: Comparison of performance in predicting language cortex activity between LLM embeddings of the original linguistic stimuli from CSD (2025) dataset (first 5 bars) and single image vision model embedding of their original COCO visual counterparts (remaining bars).
• Figure 4: Performance of SWIN vision models in predicting language cortex activity using a single image, with experiments repeated across images sorted in order of decreasing quality (defined as the cosine similarity between the sentence and the image’s CLIP embeddings).
• Figure 5: Pereira (2018) Dataset - Comparison of vision model performance in predicting language cortex activity using multiple images, with images ordered randomly versus in order of decreasing quality. Averaging more images helps in the case of random ordering, consistent with averaging the noise from non-ideal images. Adding more images eventually hurts in the case of ordered images, where eventually, less useful images are being incorporated.