Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Diverging Transformer Predictions for Human Sentence Processing: A Comprehensive Analysis of Agreement Attraction Effects

Titus von der Malsburg, Sebastian Padó

Abstract

Transformers underlie almost all state-of-the-art language models in computational linguistics, yet their cognitive adequacy as models of human sentence processing remains disputed. In this work, we use a surprisal-based linking mechanism to systematically evaluate eleven autoregressive transformers of varying sizes and architectures on a more comprehensive set of English agreement attraction configurations than prior work. Our experiments yield mixed results: While transformer predictions generally align with human reading time data for prepositional phrase configurations, performance degrades significantly on object-extracted relative clause configurations. In the latter case, predictions also diverge markedly across models, and no model successfully replicates the asymmetric interference patterns observed in humans. We conclude that current transformer models do not explain human morphosyntactic processing, and that evaluations of transformers as cognitive models must adopt rigorous, comprehensive experimental designs to avoid spurious generalizations from isolated syntactic configurations or individual models.

Diverging Transformer Predictions for Human Sentence Processing: A Comprehensive Analysis of Agreement Attraction Effects

Abstract

Transformers underlie almost all state-of-the-art language models in computational linguistics, yet their cognitive adequacy as models of human sentence processing remains disputed. In this work, we use a surprisal-based linking mechanism to systematically evaluate eleven autoregressive transformers of varying sizes and architectures on a more comprehensive set of English agreement attraction configurations than prior work. Our experiments yield mixed results: While transformer predictions generally align with human reading time data for prepositional phrase configurations, performance degrades significantly on object-extracted relative clause configurations. In the latter case, predictions also diverge markedly across models, and no model successfully replicates the asymmetric interference patterns observed in humans. We conclude that current transformer models do not explain human morphosyntactic processing, and that evaluations of transformers as cognitive models must adopt rigorous, comprehensive experimental designs to avoid spurious generalizations from isolated syntactic configurations or individual models.
Paper Structure (25 sections, 1 equation, 2 figures, 12 tables)

This paper contains 25 sections, 1 equation, 2 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Study Design.
  5. Simulation.
  6. LLM choice.
  7. Data Analysis.
  8. Experiment 1: Prepositional Modifier Construction
  9. Experimental Design
  10. Results
  11. Experiment 2: Object-Extracted Relative Clause Construction
  12. Experimental Design
  13. Results
  14. Discussion and Conclusions
  15. Regression Analyses Testing Singular-Plural Asymmetry
  16. ...and 10 more sections

Figures (2)

  • Figure 1: Surprisal predictions across models and conditions for Experiment 1 (prepositional phrases). Human performance is schematically represented in the first panel. Each sub-panel shows the effect of attraction in one of the four relevant configurations (see Table \ref{['tab:exp1-materials']}). Results for singular subjects (sg) in sub-panels 1–2; for plural (pl) in sub-panels 3–4. Means are geometric and confidence intervals 95%.
  • Figure 2: Surprisal predictions across models and conditions for Experiment 2 (relative clauses). Human performance is schematically represented in the first panel. Each sub-panel shows the effect of attraction in one of the four relevant configurations (see Table \ref{['tab:exp2-materials']}). Results for singular RC-subjects (sg) in sub-panels 1–2; for plural (pl) in sub-panels 3–4. Means are geometric and confidence intervals 95%.