Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Navigating Brain Language Representations: A Comparative Analysis of Neural Language Models and Psychologically Plausible Models

Yunhao Zhang, Shaonan Wang, Xinyi Dong, Jiajun Yu, Chengqing Zong

TL;DR

The paper investigates whether neural language models (NLMs) genuinely best predict brain language representations or if psychologically plausible models (PPMs) can do so more effectively. It compares NLMs and PPMs across eight multi-modal cognitive datasets in English and Chinese, at word- and discourse-levels, using fMRI and eye-tracking data and region/voxel-level brain encoding analyses. The results show that PPMs, especially Embodied-Based Models (EBMs) and Network-Topological Models (NTMs), outperform NLMs in brain encoding across modalities and languages, challenging the view of universal NLM supremacy. These findings highlight distinct brain-region encoding patterns linked to specific representation types and suggest cross-language generality, informing model choice for cognitive neuroscience and language processing research.

Abstract

Neural language models, particularly large-scale ones, have been consistently proven to be most effective in predicting brain neural activity across a range of studies. However, previous research overlooked the comparison of these models with psychologically plausible ones. Moreover, evaluations were reliant on limited, single-modality, and English cognitive datasets. To address these questions, we conducted an analysis comparing encoding performance of various neural language models and psychologically plausible models. Our study utilized extensive multi-modal cognitive datasets, examining bilingual word and discourse levels. Surprisingly, our findings revealed that psychologically plausible models outperformed neural language models across diverse contexts, encompassing different modalities such as fMRI and eye-tracking, and spanning languages from English to Chinese. Among psychologically plausible models, the one incorporating embodied information emerged as particularly exceptional. This model demonstrated superior performance at both word and discourse levels, exhibiting robust prediction of brain activation across numerous regions in both English and Chinese.

Navigating Brain Language Representations: A Comparative Analysis of Neural Language Models and Psychologically Plausible Models

TL;DR

The paper investigates whether neural language models (NLMs) genuinely best predict brain language representations or if psychologically plausible models (PPMs) can do so more effectively. It compares NLMs and PPMs across eight multi-modal cognitive datasets in English and Chinese, at word- and discourse-levels, using fMRI and eye-tracking data and region/voxel-level brain encoding analyses. The results show that PPMs, especially Embodied-Based Models (EBMs) and Network-Topological Models (NTMs), outperform NLMs in brain encoding across modalities and languages, challenging the view of universal NLM supremacy. These findings highlight distinct brain-region encoding patterns linked to specific representation types and suggest cross-language generality, informing model choice for cognitive neuroscience and language processing research.

Abstract

Neural language models, particularly large-scale ones, have been consistently proven to be most effective in predicting brain neural activity across a range of studies. However, previous research overlooked the comparison of these models with psychologically plausible ones. Moreover, evaluations were reliant on limited, single-modality, and English cognitive datasets. To address these questions, we conducted an analysis comparing encoding performance of various neural language models and psychologically plausible models. Our study utilized extensive multi-modal cognitive datasets, examining bilingual word and discourse levels. Surprisingly, our findings revealed that psychologically plausible models outperformed neural language models across diverse contexts, encompassing different modalities such as fMRI and eye-tracking, and spanning languages from English to Chinese. Among psychologically plausible models, the one incorporating embodied information emerged as particularly exceptional. This model demonstrated superior performance at both word and discourse levels, exhibiting robust prediction of brain activation across numerous regions in both English and Chinese.
Paper Structure (16 sections, 4 figures, 1 table)

This paper contains 16 sections, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Evaluating Neural Language Models with Cognitive Data
  4. Psychologically Plausible Models
  5. Cognitive Datasets
  6. Method
  7. Neural Language Models
  8. Psychologically Plausible Models
  9. Encoding Brain Activation
  10. Result and Discussion
  11. Comparison between Neural Language Models and Psychologically Plausible Models
  12. Comparison within Psychologically Plausible Models
  13. Comparison within Neural Language Models
  14. Encoding Performance on Each Voxel
  15. Conclusion
  16. ...and 1 more sections

Figures (4)

  • Figure 1: [a] Stimuli examples from four paradigms: picture, sentence, and word cloud paradigms are employed in word-level fMRI, while the audio paradigm is used in discourse-level fMRI. Sentence paradigm is utilized in eye-tracking data. [b] Neural encoding method using representations generated by neural language models and psychologically plausible models.
  • Figure 2: Pearson correlation coefficients between predicted and true values were computed for English and Chinese in word-level fMRI, discourse-level fMRI, and eye-tracking data using both NLMs and PPMs. Results are averaged across all subjects. To facilitate comparison, we average results from the picture paradigm, sentence paradigm, and word cloud paradigm to obtain the English word-level results. As for context-aware models, we select the layer with the best performance.
  • Figure 3: Mean Pearson correlation coefficients were calculated for each layer of context-aware models in English and Chinese word-level and discourse-level fMRI data across the entire brain. A circle marks the layer with the best encoding performance for each model.
  • Figure 4: Distribution of top-performing model for each voxel across the entire brain in English (left) and Chinese (right) discourse-level fMRI. ROIs related to semantic processing are marked.