Incremental Sentence Processing Mechanisms in Autoregressive Transformer Language Models

TL;DR

We study how autoregressive transformer LMs process language incrementally using garden-path sentences as a probe. The approach combines sparse autoencoders to extract interpretable features and causal interventions (AtP-IG) to test their functional role, along with structural probes to assess multiple readings. Key findings show that LMs rely on a mix of syntactic features and heuristics, hold simultaneous representations of multiple readings, and generally do not repair or reanalyze prior interpretations when faced with disambiguating evidence. This mechanistic view reveals that incremental processing in LMs is not solely driven by linguistically faithful structure but also by distributional cues, with implications for interpretability and robustness of language models in syntactically ambiguous contexts.

Abstract

Autoregressive transformer language models (LMs) possess strong syntactic abilities, often successfully handling phenomena from agreement to NPI licensing. However, the features they use to incrementally process language inputs are not well understood. In this paper, we fill this gap by studying the mechanisms underlying garden path sentence processing in LMs. We ask: (1) Do LMs use syntactic features or shallow heuristics to perform incremental sentence processing? (2) Do LMs represent only one potential interpretation, or multiple? and (3) Do LMs reanalyze or repair their initial incorrect representations? To address these questions, we use sparse autoencoders to identify interpretable features that determine which continuation - and thus which reading - of a garden path sentence the LM prefers. We find that while many important features relate to syntactic structure, some reflect syntactically irrelevant heuristics. Moreover, while most active features correspond to one reading of the sentence, some features correspond to the other, suggesting that LMs assign weight to both possibilities simultaneously. Finally, LMs do not re-use features from garden path sentence processing to answer follow-up questions.

Figures (16)

• Figure 1: Overview. We use sparse autoencoders to decompose model activations into a discrete set of human-interpretable components (features). We score each feature by its causal contribution to continuations associated with each reading of a garden path sentence. We manually interpret the top-scoring features and causally verify their functional role in the network by targetedly up- or downweighting them to change the model's preferred reading.
• Figure 2: Mean difference in probability of tokens corresponding to garden path (","/".") and non-garden-path ("was") readings of the input for Pythia-70m, grouped by garden path structure. Error bars indicate the standard error of the mean. Inputs are either ambiguous, or compatible with only a garden-path or non-garden-path reading. GP tokens are more likely given GP inputs; non-GP are more likely with non-GP inputs. In ambiguous cases, Pythia-70m prefers the GP reading, except for on NP/S.
• Figure 3: Pythia-70m's feature circuit for processing NP/Z garden path sentences. We group features by their functional role in the circuit and display the number of features in each group. Red features have negative scores and promote the garden path reading; blue features, with positive scores, do the opposite. Unlabeled early-layer features are word detectors. Many late-layer features encode syntactic features, whereas early-layer features largely consist of word detectors and heuristics.
• Figure 4: Mean difference in probability of GP and non-GP continuations under interventions for Pythia-70m. Error bars indicate the standard error of the mean. Interventions on interpretable features reverse model behavior, as expected; random interventions do nothing.
• Figure 5: Mean probe action probability across layers. GEN corresponds to the non-GP reading, and LEFT-ARC to the GP reading (RIGHT-ARC is implausible). NP/Z sentences elicit primarily LEFT-ARC; NP/S elicits GEN. Both valid readings always receive non-zero probability.