In-Context Learning Without Copying

TL;DR

The paper investigates whether inductive copying via induction heads is necessary for in-context learning (ICL) in transformers. By introducing Hapax, a loss-masking regime that omits gradients for positions predictable by induction heads, the authors suppress inductive copying while training 1B models on the Pile. Hapax reduces exact copying behaviors and induction-head activity, yet preserves or even improves abstractive ICL performance on many tasks, including translation, and improves fluency, suggesting abstractive ICL can emerge without reliance on induction copying. Mechanistic analyses and cross-checkpoint experiments indicate weaker induction circuits but reveal that prefix-matching patterns can form due to prior-token information, challenging the notion that induction copying is causally essential for abstractive ICL and highlighting the value of shaping data distributions to study learning dynamics.

Abstract

Induction heads are attention heads that perform inductive copying by matching patterns from earlier context and copying their continuations verbatim. As models develop induction heads, they often experience a sharp drop in training loss, a phenomenon cited as evidence that induction heads may serve as a prerequisite for more complex in-context learning (ICL) capabilities. In this work, we ask whether transformers can still acquire ICL capabilities when inductive copying is suppressed. We propose Hapax, a setting where we omit the loss contribution of any token that can be correctly predicted by induction heads. Despite a significant reduction in inductive copying, performance on abstractive ICL tasks (i.e., tasks where the answer is not contained in the input context) remains comparable and surpasses the vanilla model on 13 of 21 tasks, even though 31.7\% of tokens are omitted from the loss. Furthermore, our model achieves lower loss values on token positions that cannot be predicted correctly by induction heads. Mechanistic analysis further shows that models trained with Hapax develop fewer and weaker induction heads but still preserve ICL capabilities. Taken together, our findings indicate that inductive copying is not essential for learning abstractive ICL mechanisms.

Figures (31)

• Figure 1: Demonstration of the induction circuit. Previous token heads allow each token to store which token came previously. Induction heads do a match-and-copy operation to reproduce the subsequence that appeared earlier in the context.
• Figure 2: An overview of Hapax training regime. To suppress inductive copying, we introduce Hapax where positions predictable by induction heads within a context window do not contribute to the loss (gray positions). This discourages n-gram repetitions in the training distribution, allowing us to control the strength of inductive copying and observe its effect on in context learning.
• Figure 3: Repetition performance in both random token and natural text settings. (a) Hapax models struggle with repeating random sequences of tokens, a task that is solvable with induction circuits. (b) For natural text repetition, Hapax models actively suppress copying as training progresses. Accuracy is measured over 1000 randomly generated samples with $s = 25$.
• Figure 4: Word-level translation performance across all 3 models. We observe that both Hapax and Thresholded-Hapax increases the translation performance. This indicates that reduction in inductive copying does not hinder the model's ability to learn translation. The translations are from a given source language into English in order to keep the number of predicted tokens same across different languages. Error bars indicate 95% confidence intervals. $\dagger$ denotes statistical significant difference ($p < 0.05$) with vanilla model accuracy according to McNemar's test.
• Figure 5: Comparison of token-loss metrics and positional loss values across conditions. The Hapax model exhibits lower loss values and better TLD scores for token positions that cannot be predicted by induction. However, it receives a negative score when we consider all samples. This indicates that the loss-dependent metrics mostly capture the gains from exact copying strategy but does have implications for non-extractive samples. The metrics are calculated on randomly sampled data from the validation dataset.