The Coverage Principle: How Pre-Training Enables Post-Training
Fan Chen, Audrey Huang, Noah Golowich, Sadhika Malladi, Adam Block, Jordan T. Ash, Akshay Krishnamurthy, Dylan J. Foster
TL;DR
This work introduces the Coverage Principle, arguing that pre-training via next-token prediction shapes downstream success not just through cross-entropy, but through a tail-aware coverage profile $Cov_N(\pi_D\|\hat{\pi})$. It shows that coverage generalizes faster than cross-entropy and can predict downstream rewards for post-training methods like Best-of-N more reliably, with fewer horizon-related distortions. The authors provide theoretical results, including a generalization bound for maximum likelihood that decomposes into fine- and coarse-grained terms, and show that SGD can be improved via gradient normalization and test-time strategies. They further propose practical interventions (test-time decoding, tournament selection, on-policy generation) to enhance coverage and discuss robustness to misspecification and extensions to convex model classes. Overall, the coverage framework offers a principled lens to connect pre-training objectives with post-training efficacy and suggests concrete optimization and selection strategies to improve downstream performance.
Abstract
Language models demonstrate remarkable abilities when pre-trained on large text corpora and fine-tuned for specific tasks, but how and why pre-training shapes the success of the final model remains poorly understood. Notably, although pre-training success is often quantified by cross-entropy loss, cross-entropy can be a poor predictor of downstream performance. Instead, we provide a theoretical perspective on this relationship through the lens of \emph{coverage}, which quantifies the probability mass the pre-trained model places on high-quality responses and which is necessary and sufficient for post-training and test-time scaling methods such as Best-of-N to succeed. Our main results develop an understanding of \emph{the coverage principle}, a phenomenon whereby next-token prediction (more generally, maximum likelihood) implicitly optimizes toward a model with good coverage. In particular, we uncover a mechanism that explains the power of coverage in predicting downstream performance: \emph{coverage generalizes faster than cross-entropy}, avoiding spurious dependence on problem-dependent parameters such as the sequence length. We also study practical algorithmic interventions with provable benefits for improving coverage, including (i) model/checkpoint selection procedures, (ii) gradient normalization schemes, and (iii) test-time decoding strategies.