An Information-Theoretic Analysis of In-Context Learning
Hong Jun Jeon, Jason D. Lee, Qi Lei, Benjamin Van Roy
TL;DR
This work develops a general information-theoretic framework for meta-learning from sequential data, decomposing the optimal Bayesian error into irreducible, meta-learning, and intra-task components. The authors prove linear decay of error with respect to both the number of sequences and their lengths, without relying on stability or mixing assumptions, and apply the theory to in-context learning in transformers and to sparse mixtures of transformers. Key contributions include exact Bayesian error characterizations, rate-distortion based bounds, and concrete results for deep transformer architectures and mixture models that illuminate how ICL can succeed with limited in-context data. The framework offers a principled lens for understanding ICL and sequential meta-learning in non-iid settings, with implications for designing and analyzing large language models.
Abstract
Previous theoretical results pertaining to meta-learning on sequences build on contrived assumptions and are somewhat convoluted. We introduce new information-theoretic tools that lead to an elegant and very general decomposition of error into three components: irreducible error, meta-learning error, and intra-task error. These tools unify analyses across many meta-learning challenges. To illustrate, we apply them to establish new results about in-context learning with transformers. Our theoretical results characterizes how error decays in both the number of training sequences and sequence lengths. Our results are very general; for example, they avoid contrived mixing time assumptions made by all prior results that establish decay of error with sequence length.