Illusion or Algorithm? Investigating Memorization, Emergence, and Symbolic Processing in In-Context Learning

TL;DR

This work interrogates in-context learning (ICL) in large Transformers trained on web-scale data, asking whether ICL arises from memorization, generalization, or a symbolic algorithm, and how it develops during pre-training. By introducing a suite of pattern-based, random-token tasks and leveraging the full Pythia scaling suite, the authors demonstrate that ICL is not mere memorization but also not a universal symbolic algorithm; performance depends on token frequency and task configuration, indicating a mix of generalization and statistics. They reveal a gradual, scalable development of ICL across pre-training, with a scaling law governing the emergence of more complex ICL capabilities, and they connect ICL development to internal subspace formation in the residual stream using the SUDA method, showing early emergence of dominant subspaces around $\sim$1000 steps and a peak in strong subspaces near $\sim$20{,}000 steps. The findings have implications for AI safety and model design, suggesting that ICL is a tractable, measurable phenomenon that can be analyzed and guided rather than an unpredictable emergent capability. Overall, the paper advances mechanistic understanding of ICL by tying behavioral generalization to internal subspace dynamics while acknowledging token statistics as a fundamental constraint on off-distribution generalization.

Abstract

Large-scale Transformer language models (LMs) trained solely on next-token prediction with web-scale data can solve a wide range of tasks after seeing just a few examples. The mechanism behind this capability, known as in-context learning (ICL), remains both controversial and poorly understood. Some studies argue that it is merely the result of memorizing vast amounts of data, while others contend that it reflects a fundamental, symbolic algorithmic development in LMs. In this work, we introduce a suite of investigative tasks and a novel method to systematically investigate ICL by leveraging the full Pythia scaling suite, including interim checkpoints that capture progressively larger amount of training data. By carefully exploring ICL performance on downstream tasks and simultaneously conducting a mechanistic analysis of the residual stream's subspace, we demonstrate that ICL extends beyond mere "memorization" of the training corpus, yet does not amount to the implementation of an independent symbolic algorithm. Our results also clarify several aspects of ICL, including the influence of training dynamics, model capabilities, and elements of mechanistic interpretability. Overall, our work advances the understanding of ICL and its implications, offering model developers insights into potential improvements and providing AI security practitioners with a basis for more informed guidelines.

Figures (13)

• Figure 1: LM performance on the lsc task deteriorates as tokens are sampled from ranges with larger indices. The figure shows the final checkpoint performance of various sufficiently large Pythia models (410M+) on the pattern repeating task, with tokens drawn from different index ranges. The maximum and minimum performance on the same task but with Brown Corpus tokens are highlighted by the two vertical lines.
• Figure 2: ICL performance varies across tasks and configurations designed to control the "difficulty" of tasks.
• Figure 3: Literal sequence copying ( lsc) ability develops differently for in- and off-distribution generalization in the early stages of pre-training. In-distribution generalization develops earlier and more rapidly, whereas off-distribution generalization tends to develop later and more slowly.
• Figure 4: The early development of Pythia-12b's lsc ability measured in log probability, a continuous metric. The result supports the same findings.
• Figure 5: The dispersion between in- and off-distribution generalization ability. For all model sizes, we observe that in-distribution generalization develops earlier and faster. The gap decreases with more training but stabilizes at a stable, positive level.