Curriculum-Guided Layer Scaling for Language Model Pretraining

TL;DR

Curriculum-Guided Layer Scaling (CGLS) introduces a compute-efficient pretraining framework that synchronizes increasing data difficulty with progressive model depth. By coupling curriculum learning with staged layer expansion and a two-phase training protocol, CGLS preserves prior representations while growing capacity, yielding improved generalization on knowledge-intensive and reasoning benchmarks across GPT-2-Small and Llama-3.2-1B scales, and extending benefits to Chinchilla-optimal budgets and domain-adaptive code pretraining. The findings demonstrate that jointly increasing data complexity and depth can outperform compute-matched baselines under fixed FLOPs, with robust gains across seeds and curriculum signals, and suggest broad applicability to larger models and multimodal settings.

Abstract

As the cost of pretraining large language models grows, there is continued interest in strategies to improve learning efficiency during this core training stage. Motivated by cognitive development, where humans gradually build knowledge as their brains mature, we propose Curriculum-Guided Layer Scaling (CGLS), a framework for compute-efficient pretraining that synchronizes increasing data difficulty with model growth through progressive layer stacking (i.e. gradually adding layers during training). At the 100M parameter scale, using a curriculum transitioning from synthetic short stories to general web data, CGLS outperforms baseline methods on the question-answering benchmarks PIQA and ARC. Pretraining at the 1.2B scale, we stratify the DataComp-LM corpus with a DistilBERT-based classifier and progress from general text to highly technical or specialized content. Our results show that progressively increasing model depth alongside sample difficulty leads to better generalization and zero-shot performance on various downstream benchmarks. Altogether, our findings demonstrate that CGLS unlocks the potential of progressive stacking, offering a simple yet effective strategy for improving generalization on knowledge-intensive and reasoning tasks.

Figures (5)

• Figure 1: Curriculum-Guided Layer Scaling (CGLS) is a new paradigm for compute-efficient language model pretraining that grows data complexity and model depth in tandem. We illustrate CGLS for a Llama-3.2-1B scale model with four training stages. Training begins with an 8-layer model on a data split consisting equally of data from all levels (high-school, undergraduate, and graduate). The learned weights from this stage are transferred to a larger 10-layer model, freezing the pretrained weights and training the new layers on a small, balanced data split for better initialization. The entire model is then unfrozen and pretrained on the more difficult data split. This process is repeated until the target model scale is reached.
• Figure 2: Results for 1D hyperparameter sweeps over the starting model depth and stage-wise training budgets for the 1B-parameter model, trained on 2.5B tokens from DCLM. The average across the benchmarks is denoted with a dashed line.
• Figure 3: Model perplexity at GPT-2-Small parameter count across various individual subsets of The Pile, for three CGLS configurations and the strongest two baselines. Lower perplexity scores indicate better performance. CGLS consistently outperforms the baselines, particularly on complex knowledge-intensive datasets such as ArXiv, PubMed, and NIH.
• Figure 4: Model perplexity across various individual subsets of The Pile for all baselines versus CGLS for Llama-3.2-1B. Lower perplexity scores indicate better performance. Across all subsets, the randomized baseline achieves the lowest perplexity, with progressive scaling models outperforming MIDAS and layer scaling without a data curriculum.
• Figure 5: Validation performance at each training stage for all methods. BL refers to the baseline, with R denoting the randomized variant, and C the curricularized variant.