Preference Curriculum: LLMs Should Always Be Pretrained on Their Preferred Data

TL;DR

This work introduces Perplexity Difference PD as a metric to quantify how differently a sample is fitted by weak versus strong model checkpoints, and leverages it in a PD-based Preference Curriculum (PDPC) to continuously organize pretraining data without interrupting training. By offline approximating data preferences through two reference models and an S-shaped mixing function, PDPC progressively shifts from low-PD to high-PD data to maximize learning on challenging samples while preserving batch diversity. Empirical results on 1.3B and 3B models show PDPC consistently outperforms baselines, with the 3B model trained on 1T tokens achieving up to 8.1% higher accuracy on MMLU/CMMLU and 4.1% on average across benchmarks. The approach emphasizes data diversity, stability, and data-source/quality considerations, suggesting practical benefits for scaling LLM pretraining with model-aware curricula.

Abstract

Large language models (LLMs) generally utilize a consistent data distribution throughout the pretraining process. However, as the model's capability improves, it is intuitive that its data preferences dynamically change, indicating the need for pretraining with different data at various training stages. To achieve it, we propose the Perplexity Difference (PD) based Preference Curriculum learning (PDPC) framework, which always perceives and uses the data preferred by LLMs to train and boost them. First, we introduce the PD metric to quantify the difference in how challenging a sample is for weak versus strong models. Samples with high PD are more challenging for weak models to learn and are more suitable to be arranged in the later stage of pretraining. Second, we propose the preference function to approximate and predict the data preference of the LLM at any training step, so as to complete the arrangement of the dataset offline and ensure continuous training without interruption. Experimental results on 1.3B and 3B models demonstrate that PDPC significantly surpasses baselines. Notably, the 3B model trained on 1T tokens achieves an increased average accuracy of over 8.1% across MMLU and CMMLU.

Figures (17)

• Figure 1: PD-based Preference Curriculum Framework.
• Figure 2: Left: Preference Functions and Right: a comparison of different data sampling methods, the regions highlighted in green represent the preferred data.
• Figure 4: Ablation on PD calculation methods.
• Figure 5: (a) Relative test loss and (b) gradient norm during model training of 1B model. (c)Training loss and (d) gradient norm during 3B model training.
• Figure 6: Data distribution across different sources.