Perplexity-Aware Data Scaling Law: Perplexity Landscapes Predict Performance for Continual Pre-training

TL;DR

This paper tackles the inefficiency of continual pre-training (CPT) by proposing perplexity landscapes as a diagnostic of domain data usefulness and introducing a perplexity-aware data scaling law. The authors relate domain perplexity statistics, specifically the mean $\mu$ and variance $\sigma$ of perplexity, to final CPT performance, enabling adaptive data subset selection through a distance-to-optimum principle. The core contributions include the perplexity landscape diagnostic, a scalable scaling law linking $\mu$, $\sigma$, and data size $D$ to loss $\hat{L}$, and a greedy DOS data-selection algorithm that targets the optimum perplexity region. Empirically, the approach yields near-optimal subsets and improves medical-domain benchmarks while preserving general-domain performance, demonstrating significant data efficiency gains for domain adaptation.

Abstract

Continual Pre-training (CPT) serves as a fundamental approach for adapting foundation models to domain-specific applications. Scaling laws for pre-training define a power-law relationship between dataset size and the test loss of an LLM. However, the marginal gains from simply increasing data for CPT diminish rapidly, yielding suboptimal data utilization and inefficient training. To address this challenge, we propose a novel perplexity-aware data scaling law to establish a predictive relationship between the perplexity landscape of domain-specific data and the test loss. Our approach leverages the perplexity derived from the pre-trained model on domain data as a proxy for estimating the knowledge gap, effectively quantifying the informational perplexity landscape of candidate training samples. By fitting this scaling law across diverse perplexity regimes, we enable adaptive selection of high-utility data subsets, prioritizing content that maximizes knowledge absorption while minimizing redundancy and noise. Extensive experiments demonstrate that our method consistently identifies near-optimal training subsets and achieves superior performance on both medical and general-domain benchmarks.

Figures (7)

• Figure 1: Interdependence between loss and $\mu$ ($\sigma$).
• Figure 2: Fitted curves for scaling law based on Qwen3-14B-Base.
• Figure 3: Performance changes with varying PPL distributions and Parameters.
• Figure 4: Validation curves of scaling law.
• Figure 5: Visualizations for perplexity landscape on Qwen3-14B-base model.