The Inductive Bias of In-Context Learning: Rethinking Pretraining Example Design
Yoav Levine, Noam Wies, Daniel Jannai, Dan Navon, Yedid Hoshen, Amnon Shashua
TL;DR
This paper formalizes the in-context bias arising from how training data is chunked into examples for neural language models, showing that dependencies between sentences seen together in a training example are inherently easier to model than dependencies across different examples. It introduces the epsilon-separation rank and a sequential variant to measure cross-example expressivity, proving an expressivity gap that favors in-context interactions and quantifying a depth deficit tied to the learning rate. Building on this theory, the authors present kNN-TAPT and kNN-Pretraining, two pretraining schemes that deliberately group semantically related sentences in-context, demonstrating improved sentence representations and open-domain QA performance. The work provides both a rigorous theoretical lens on pretraining dynamics and practical guidelines for designing training examples to enhance natural language understanding. Overall, it points to new, bias-aware preprocessing strategies that can enhance self-improving representation learning in large language models.
Abstract
Pretraining Neural Language Models (NLMs) over a large corpus involves chunking the text into training examples, which are contiguous text segments of sizes processable by the neural architecture. We highlight a bias introduced by this common practice: we prove that the pretrained NLM can model much stronger dependencies between text segments that appeared in the same training example, than it can between text segments that appeared in different training examples. This intuitive result has a twofold role. First, it formalizes the motivation behind a broad line of recent successful NLM training heuristics, proposed for the pretraining and fine-tuning stages, which do not necessarily appear related at first glance. Second, our result clearly indicates further improvements to be made in NLM pretraining for the benefit of Natural Language Understanding tasks. As an example, we propose "kNN-Pretraining": we show that including semantically related non-neighboring sentences in the same pretraining example yields improved sentence representations and open domain question answering abilities. This theoretically motivated degree of freedom for pretraining example design indicates new training schemes for self-improving representations.