Characterizing Learning Curves During Language Model Pre-Training: Learning, Forgetting, and Stability

TL;DR

The paper investigates how language models acquire knowledge during pre-training by extracting $1M$ token-in-context learning curves across five autoregressive runs. It introduces metrics to quantify final surprisal, within-run variability, AoA, and forgettability, and links these to simple text features such as token frequency and $5$-gram probability. The results reveal consistent, non-random fluctuations in token learning, with frequent and high-probability $n$-grams learned faster and more stably, while tail $n$-grams show greater instability and forgetting; across runs, token learning curves are broadly similar. The findings support a view of sequential learning where early $n$-gram learning lays the foundation for later refinement with longer contexts and more nuanced linguistic capabilities, with important implications for robust deployment in domain-specific settings and for guiding pre-training curricula.

Abstract

How do language models learn to make predictions during pre-training? To study this, we extract learning curves from five autoregressive English language model pre-training runs, for 1M unseen tokens in context. We observe that the language models generate short repetitive phrases before learning to generate longer and more coherent text. We also find that individual tokens often exhibit sudden increases or decreases in loss that are surprisingly consistent across pre-training runs. To better understand these fluctuations, we quantify the final surprisal, within-run variability, age of acquisition, forgettability, and cross-run variability of learning curves for individual tokens in context. More frequent tokens reach lower final surprisals, exhibit less variability within and across pre-training runs, are learned earlier, and are less likely to be "forgotten" during pre-training. Higher n-gram probabilities further accentuate these effects. Independent of the target token, shorter and more frequent contexts correlate with marginally more stable and quickly acquired predictions. Based on our results, we argue for the existence of sequential learning dependencies between different model capabilities, and we characterize language model learning as early n-gram learning before gradual refinement of tail n-gram predictions.

Figures (4)

• Figure 1: Learning curves for three evaluation examples from the OSCAR dataset during one pre-training run. Colored lines are fitted GAM curves.
• Figure 2: Mean pairwise correlation between model surprisals for different pre-training runs, at different pre-training steps. Shaded regions indicate five standard deviations from the mean. Vertical lines indicate the pre-training steps where model surprisals are maximally correlated with $n$-gram surprisals.
• Figure 3: Learning curves for two evaluation examples from the OSCAR dataset with high forgettability scores, for the five pre-training runs. Purple lines are fitted GAM curves, one per pre-training run.
• Figure 4: Loss curves (mean surprisal) for all five pre-training runs. Loss curves are nearly identical across runs. To align with other figures, pre-training steps are reported in log10.