How does GPT-2 Predict Acronyms? Extracting and Understanding a Circuit via Mechanistic Interpretability

TL;DR

The paper tackles the interpretability gap for multi-token predictions in transformers by using mechanistic interpretability and activation patching to identify a dedicated 8-head circuit in GPT-2 Small that predicts three-letter acronyms. It introduces the concept of Letter Mover Heads, which copy the capital-letter content from the previous word's token to the next acronym letter, and demonstrates that these heads rely on positional information provided by the causal mask—particularly the attention to BOS—rather than solely on positional embeddings. Ablation shows the 8-head circuit is sufficient to reproduce the task performance, suggesting a modular, reusable mechanism for multi-token predictions. The study provides a concrete foundation for analyzing more complex behaviors in larger models, with implications for safety, reliability, and interpretability of AI systems in multi-token tasks.

Abstract

Transformer-based language models are treated as black-boxes because of their large number of parameters and complex internal interactions, which is a serious safety concern. Mechanistic Interpretability (MI) intends to reverse-engineer neural network behaviors in terms of human-understandable components. In this work, we focus on understanding how GPT-2 Small performs the task of predicting three-letter acronyms. Previous works in the MI field have focused so far on tasks that predict a single token. To the best of our knowledge, this is the first work that tries to mechanistically understand a behavior involving the prediction of multiple consecutive tokens. We discover that the prediction is performed by a circuit composed of 8 attention heads (~5% of the total heads) which we classified in three groups according to their role. We also demonstrate that these heads concentrate the acronym prediction functionality. In addition, we mechanistically interpret the most relevant heads of the circuit and find out that they use positional information which is propagated via the causal mask mechanism. We expect this work to lay the foundation for understanding more complex behaviors involving multiple-token predictions.

Figures (24)

• Figure 1: Patching the residual stream at every position and before every layer (corrupting the current word).
• Figure 2: Patching the output of attention heads for every iteration (corrupting the current word).
• Figure 3: Average probability paid from A(i-1) to the previous token positions for head 8.11.
• Figure 4: Patching the residual stream (corrupting previous words).
• Figure 5: Patching the output of attention heads for every iteration at position Ci (corrupting the previous words).