Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Curriculum Learning for LLM Pretraining: An Analysis of Learning Dynamics

Mohamed Elgaar, Hadi Amiri

TL;DR

This work analyzes how pretraining data order shapes learning dynamics in large language models by training Pythia variants (14M–410M parameters) on 300B tokens under three linguistically motivated curricula and comparing to Random ordering. It finds that all orderings follow a shared sequence of latent learning phases, and curricula mainly affect data exposure within phases, yielding greater stability and modest accuracy gains in smaller models through reduced gradient noise and less spectral saturation of the output head. A theoretical framework links difficulty pacing to bounded gradient variance, explaining why curricula stabilize late-stage optimization, especially when model capacity is insufficient to avoid softmax bottlenecks. Gains shrink with scale, suggesting curricula are most beneficial for capacity-constrained regimes and guiding practical, phase-aware curriculum designs, including adaptive and multi-stage strategies for broader applicability. The results offer actionable insights for robust, compute-efficient pretraining, highlighting how within-phase data exposure, not phase creation, underpins stability improvements.

Abstract

Curriculum learning changes the order of pre-training data, but it remains unclear whether it changes the learning trajectory or mainly reorders exposure over a fixed trajectory. We train Pythia models (14M-410M parameters) for 300B tokens under three linguistically motivated curricula-Age-of-Acquisition, word frequency, and Verb Variation (VV)-and compare each against Random ordering; at 1B parameters we compare Random and VV. Across orderings, training follows a shared sequence of latent phases, while curricula mainly change within-phase data exposure. In smaller models (up to 160M parameters), Random ordering exhibits higher gradient noise and stronger late-training output-head spectral saturation, alongside lower final accuracy; curricula reduce both effects at matched compute. At larger scales, saturation differences are smaller and curriculum gains shrink. We formalize the link between difficulty pacing and optimization stability in an idealized analysis based on gradient-variance control, and our results point to a practical takeaway: curricula help by stabilizing within-phase optimization rather than by creating new phases.

Curriculum Learning for LLM Pretraining: An Analysis of Learning Dynamics

TL;DR

This work analyzes how pretraining data order shapes learning dynamics in large language models by training Pythia variants (14M–410M parameters) on 300B tokens under three linguistically motivated curricula and comparing to Random ordering. It finds that all orderings follow a shared sequence of latent learning phases, and curricula mainly affect data exposure within phases, yielding greater stability and modest accuracy gains in smaller models through reduced gradient noise and less spectral saturation of the output head. A theoretical framework links difficulty pacing to bounded gradient variance, explaining why curricula stabilize late-stage optimization, especially when model capacity is insufficient to avoid softmax bottlenecks. Gains shrink with scale, suggesting curricula are most beneficial for capacity-constrained regimes and guiding practical, phase-aware curriculum designs, including adaptive and multi-stage strategies for broader applicability. The results offer actionable insights for robust, compute-efficient pretraining, highlighting how within-phase data exposure, not phase creation, underpins stability improvements.

Abstract

Curriculum learning changes the order of pre-training data, but it remains unclear whether it changes the learning trajectory or mainly reorders exposure over a fixed trajectory. We train Pythia models (14M-410M parameters) for 300B tokens under three linguistically motivated curricula-Age-of-Acquisition, word frequency, and Verb Variation (VV)-and compare each against Random ordering; at 1B parameters we compare Random and VV. Across orderings, training follows a shared sequence of latent phases, while curricula mainly change within-phase data exposure. In smaller models (up to 160M parameters), Random ordering exhibits higher gradient noise and stronger late-training output-head spectral saturation, alongside lower final accuracy; curricula reduce both effects at matched compute. At larger scales, saturation differences are smaller and curriculum gains shrink. We formalize the link between difficulty pacing and optimization stability in an idealized analysis based on gradient-variance control, and our results point to a practical takeaway: curricula help by stabilizing within-phase optimization rather than by creating new phases.
Paper Structure (44 sections, 4 theorems, 17 equations, 6 figures, 9 tables)

This paper contains 44 sections, 4 theorems, 17 equations, 6 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Theoretical Framework for Curriculum Learning
  4. The Softmax Bottleneck and Rank Saturation (details in appendix)
  5. The Link Between Ideal Difficulty and Gradient Variance
  6. The Gradient Noise Scale
  7. Uniform Sampling and Difficulty Pacing (details in appendix)
  8. Curriculum Learning as a Stability Mechanism
  9. Mapping the framework to our curricula (difficulty score and pacing)
  10. Difficulty score $d(z)$.
  11. Pacing function $p(t)$.
  12. Pretraining Curriculum Design
  13. Linguistic Characterization of Curriculum Indices
  14. Models and Training Setup
  15. Analyzing Learning Dynamics
  16. ...and 29 more sections

Key Result

Theorem 3.2

In an idealized strongly convex setting, the stability radius of SGD scales with the effective gradient variance: sampling schemes that keep $\sigma_t^2$ bounded yield tighter stability bounds. Under uniform sampling, $\sigma_t^2$ may drift toward a high-variance regime late in training; curricula i

Figures (6)

  • Figure 1: Overview of the mechanism studied in this work. In smaller, capacity-constrained models, random data ordering can produce higher gradient noise scale, coinciding with late-stage spectral saturation of the output head. Linguistically motivated curricula (Age-of-Acquisition, Frequency, and Verb Variation) reduce gradient noise and are associated with improved spectral stability.
  • Figure 2: Singular entropy of the language modeling head versus training step. For models up to 160M parameters, Random ordering produces sharp late-stage increases in singular entropy, indicating collapse toward a spiked spectrum associated with saturation. Curriculum-based orderings maintain lower entropy, avoiding the degenerate spectral regime associated with performance degradation.
  • Figure 3: Gradient noise scale (GNS) versus training step across model sizes and orderings. Random ordering tends to produce higher GNS for smaller models (14M--70M), indicating noisier gradient estimates and less efficient optimization. At larger scales (160M--410M), the gap between Random and the curricula is smaller and less consistent.
  • Figure 4: Shared phases and within-phase dynamics for the 14M model. \ref{['fig:training_map']} shows the average accuracy trajectory for Random, Age-of-Acquisition, Frequency, and VV, with checkpoints colored by the HMM state from \ref{['fig:hmm_transitions']}. \ref{['fig:hmm_transitions']} shows the shared HMM state transition diagram learned jointly across orderings. Curricula do not change the phases, but affect the accuracy trajectory within those phases.
  • Figure 5: Pairwise ordering comparisons on BLiMP probes with consistent differences. These comparisons align with the effects discussed in the main text: VV improves wh-movement accuracy while Frequency can underperform Random on quantifier scope.
  • ...and 1 more figures

Theorems & Definitions (8)

  • Definition 3.1: Ideal Difficulty Score
  • Theorem 3.2: Curriculum Learning and Stability (informal)
  • Lemma 1.1: Inherent Performance Gap, from godeysmall
  • Definition 1.4: Pacing Function
  • Theorem 1.5: \ref{['thm:stability']}, restatement
  • Lemma 1.6: SGD Stability under Bounded Gradient Variance
  • proof : Proof of \ref{['lem:sgd_stability']}
  • proof : Proof of \ref{['thm:stability']}