Scaling Laws Revisited: Modeling the Role of Data Quality in Language Model Pretraining

TL;DR

The paper addresses the gap in scaling laws by explicitly modeling data quality in pretraining through a dimensionless parameter $Q$. It proposes a quality-aware law, $L(N,D,Q) = \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta} Q^{\gamma}} + E$, grounded in effective-sample-size and information-theoretic arguments, and provides two estimators for $Q$. Through controlled experiments on neural machine translation and causal language modeling with synthetic noise, the authors show that loss scales predictably with data quality and that higher-quality data can compensate for smaller models, yielding sublinear degradation ($\gamma<1$) in many settings. The work offers a generalizable framework to balance data curation and compute in domain-specific pretraining, enabling more efficient deployment of smaller, high-quality models in specialized applications.

Abstract

Scaling laws for language model training traditionally characterize how performance scales with model size and dataset volume. Prior work has explored architecture variants and data treatments such as dataset filtering and noise injection in language model pretraining; however, these studies have not formalized data quality within a principled scaling law. We introduce a dimensionless data-quality parameter Q, and propose a quality-aware scaling law extending the Chinchilla framework to predict loss as a joint function of model size, data volume, and data quality. The law is motivated by an effective-sample-size and information-theoretic view of noisy or redundant corpora, and it admits two practical estimators for Q: (i) a corruption rate proxy and (ii) a deficiency measure. Through synthetic experiments in neural machine translation and autoregressive modeling -- where we systematically control data quality via multiple levels of noise injection and coverage variation -- we show that loss scales predictably with data quality and that higher-quality data can substantially reduce model size and hence compute requirements. Our results demonstrate a sublinear decay of effective data with quality and robustness to moderate data corruption; out-of-sample evaluations further validate the predictive form of the law. Unlike prior empirical analyses, our work establishes an explicit, generalizable law for data quality, offering concrete guidance for balancing data curation effort and model scale in large-scale pretraining.

Figures (3)

• Figure 1: Test loss as data quality varies from 0.5 to 1 for a) machine translation and b) next token prediction.
• Figure 2: Iso loss contours for machine translation (a) and next token prediction (b).
• Figure 3: $\Delta L(Q)$ vs. $Q^{-\hat{\gamma}} - 1$ for a) NMT and b) CLM, where $\hat{\gamma}$ is the estimated $\gamma$ parameter using the Huber estimation from the scaling experiments.

Theorems & Definitions (11)

• Definition 1
• Definition 2
• Definition 3
• Lemma 1: Effective sample size under additive Gaussian noise
• Lemma 2: Effective sample size under random classification noise
• Definition 4: Quality-Aware Scaling Law
• Corollary 1
• Proposition 1: Information-theoretic justification
• proof : Proof of Lemma \ref{['lem:additive-noise']}
• proof : Proof of Lemma \ref{['lem:rcn']}