Incremental Comprehension of Garden-Path Sentences by Large Language Models: Semantic Interpretation, Syntactic Re-Analysis, and Attention

TL;DR

The paper investigates how large language models process temporarily ambiguous garden-path sentences and whether they align with human behavior in semantic interpretation, syntactic reanalysis, and attention. It uses chunked presentations of 24 garden-path items and three experimental angles—surprisal baselines, semantic interpretation tracking, and incremental parse-tree extraction, plus attention visualization—across four diverse LLMs (GPT-2, LLaMA-2, Flan-T5, RoBERTa). Key findings show partial human-like alignment, with stronger convergence when extra-syntactic cues such as a disambiguating comma are present, and RoBERTa-large and LLaMA-2 displaying notable parsing and interpretation shifts toward the correct reading. These results support the use of LLMs as scientific models of human sentence processing while highlighting limitations and the need for broader model coverage and materials to generalize the insights.

Abstract

When reading temporarily ambiguous garden-path sentences, misinterpretations sometimes linger past the point of disambiguation. This phenomenon has traditionally been studied in psycholinguistic experiments using online measures such as reading times and offline measures such as comprehension questions. Here, we investigate the processing of garden-path sentences and the fate of lingering misinterpretations using four large language models (LLMs): GPT-2, LLaMA-2, Flan-T5, and RoBERTa. The overall goal is to evaluate whether humans and LLMs are aligned in their processing of garden-path sentences and in the lingering misinterpretations past the point of disambiguation, especially when extra-syntactic information (e.g., a comma delimiting a clause boundary) is present to guide processing. We address this goal using 24 garden-path sentences that have optional transitive and reflexive verbs leading to temporary ambiguities. For each sentence, there are a pair of comprehension questions corresponding to the misinterpretation and the correct interpretation. In three experiments, we (1) measure the dynamic semantic interpretations of LLMs using the question-answering task; (2) track whether these models shift their implicit parse tree at the point of disambiguation (or by the end of the sentence); and (3) visualize the model components that attend to disambiguating information when processing the question probes. These experiments show promising alignment between humans and LLMs in the processing of garden-path sentences, especially when extra-syntactic information is available to guide processing.

Figures (8)

• Figure 1: An example demonstrating the garden-path effect. During the incremental processing of this sentence, readers initially expect deer to be the object of hunted. Upon reaching the second verb run, they realize that deer is actually the subject of the second clause. This prompts a reanalysis of the sentence to the correct interpretation. However, the misinterpretation sometimes remains active even after reanalysis, and people still verify that "the man hunted the deer".
• Figure 2: Surprisal in the first task (sentences with comma absent) across the four models.
• Figure 3: Surprisal in the second task (sentences with the disambiguating comma present) across the four models.
• Figure 4: Semantic tracking of the mis- and correct interpretation in the first task (sentences with comma absent) across the four models. Critically, the probability of misinterpretation remains high even after the point of disambiguation.
• Figure 5: Semantic tracking of the mis- and correct interpretation in the second task (sentences with the disambiguating comma present). In the presence of this extra-syntactic information, the probability of misinterpretation decreases after the point of disambiguation for LLaMA, aligning with human performance.