Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Modeling cognitive processes of natural reading with transformer-based Language Models

Bruno Bianchi, Fermín Travi, Juan E. Kamienkowski

TL;DR

This work evaluates transformer-based language models as predictors of eye-tracking measures during natural reading in Rioplatense Spanish, comparing model-derived comp-Pred with human cloze-Pred within linear mixed-effects models. It extends prior approaches by using GPT2 and LLaMA variants to generate computational predictabilities and assesses their additive and residual relationships to First Pass Reading Time. The results show transformer models explain more variance than earlier approaches but do not fully match human predictability, with cloze-Pred residual variance still present and interacting with lexical frequency. The study underscores a partial convergence between AI language modeling and human reading processes, while emphasizing the need for broader cross-language validation and deeper exploration of what these models learn about real-time language comprehension.

Abstract

Recent advances in Natural Language Processing (NLP) have led to the development of highly sophisticated language models for text generation. In parallel, neuroscience has increasingly employed these models to explore cognitive processes involved in language comprehension. Previous research has shown that models such as N-grams and LSTM networks can partially account for predictability effects in explaining eye movement behaviors, specifically Gaze Duration, during reading. In this study, we extend these findings by evaluating transformer-based models (GPT2, LLaMA-7B, and LLaMA2-7B) to further investigate this relationship. Our results indicate that these architectures outperform earlier models in explaining the variance in Gaze Durations recorded from Rioplantense Spanish readers. However, similar to previous studies, these models still fail to account for the entirety of the variance captured by human predictability. These findings suggest that, despite their advancements, state-of-the-art language models continue to predict language in ways that differ from human readers.

Modeling cognitive processes of natural reading with transformer-based Language Models

TL;DR

This work evaluates transformer-based language models as predictors of eye-tracking measures during natural reading in Rioplatense Spanish, comparing model-derived comp-Pred with human cloze-Pred within linear mixed-effects models. It extends prior approaches by using GPT2 and LLaMA variants to generate computational predictabilities and assesses their additive and residual relationships to First Pass Reading Time. The results show transformer models explain more variance than earlier approaches but do not fully match human predictability, with cloze-Pred residual variance still present and interacting with lexical frequency. The study underscores a partial convergence between AI language modeling and human reading processes, while emphasizing the need for broader cross-language validation and deeper exploration of what these models learn about real-time language comprehension.

Abstract

Recent advances in Natural Language Processing (NLP) have led to the development of highly sophisticated language models for text generation. In parallel, neuroscience has increasingly employed these models to explore cognitive processes involved in language comprehension. Previous research has shown that models such as N-grams and LSTM networks can partially account for predictability effects in explaining eye movement behaviors, specifically Gaze Duration, during reading. In this study, we extend these findings by evaluating transformer-based models (GPT2, LLaMA-7B, and LLaMA2-7B) to further investigate this relationship. Our results indicate that these architectures outperform earlier models in explaining the variance in Gaze Durations recorded from Rioplantense Spanish readers. However, similar to previous studies, these models still fail to account for the entirety of the variance captured by human predictability. These findings suggest that, despite their advancements, state-of-the-art language models continue to predict language in ways that differ from human readers.
Paper Structure (9 sections, 1 figure, 2 tables)

This paper contains 9 sections, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Data: Texts, Cloze-Pred Estimation, and Eye Movements
  4. Human Predictability (cloze-Pred)
  5. Computational Predictability (comp-Pred)
  6. Linear Mixed Models
  7. Results
  8. Discussion
  9. Limitations

Figures (1)

  • Figure 1: Experimental designs: (A) Eye movements were recorded in native Spanish readers reading 8 narrative stories. The eye movement measure (FPRT) was used as dependent variable in statistical models in R. (B) cloze-Pred for texts used in (A) was estimated from online responses by +1,000 participants in a cloze-task experiment. (C) comp-Pred for the same texts was estimated with several Language Models. (D) Statistical models ran in the present study. cloze- and comp-Preds were individually added to a baseline model. Model comparison was performed using AIC difference. (E) Diagram of expected results for cloze and comp-Pred effects after the Remef analysis.