Scaling Parameter-Constrained Language Models with Quality Data

TL;DR

This paper pretrained over 200 models of 25M to 1.5B parameters on a diverse set of sampled, synthetic data, and estimated the constants that relate text quality, model size, training tokens, and eight reasoning task accuracy scores.

Abstract

Scaling laws in language modeling traditionally quantify training loss as a function of dataset size and model parameters, providing compute-optimal estimates but often neglecting the impact of data quality on model generalization. In this paper, we extend the conventional understanding of scaling law by offering a microscopic view of data quality within the original formulation -- effective training tokens -- which we posit to be a critical determinant of performance for parameter-constrained language models. Specifically, we formulate the proposed term of effective training tokens to be a combination of two readily-computed indicators of text: (i) text diversity and (ii) syntheticity as measured by a teacher model. We pretrained over $200$ models of 25M to 1.5B parameters on a diverse set of sampled, synthetic data, and estimated the constants that relate text quality, model size, training tokens, and eight reasoning task accuracy scores. We demonstrated the estimated constants yield +0.83 Pearson correlation with true accuracies, and analyzed it in scenarios involving widely-used data techniques such as data sampling and synthesis which aim to improve data quality.

Figures (9)

• Figure 1: Correlations between text diversity scores on 1% of RefinedWeb penedo2023refinedweb. Similar to shaib2024standardizing, compression ratio (CR) correlates strongly with most other diversity metrics.
• Figure 2: This plot illustrates the impact of various data refinement techniques on the effective token count ($D_q$) as the number of tokens is scaled up. Experiments were performed with RefinedWeb penedo2023refinedweb data.
• Figure 3: Plots of revised scaling law with qualitative data measurements. Left: Plot of averaged accuracy against effective tokens $D_q$ where $D_q = D \cdot \exp(c_1 \cdot \text{Dr}(D) + c_2 \cdot \text{S}(D))$. The accuracy values are the reference values. Right: Impact of scaling factor $Q$ on both diversity and syntheticity. Interestingly, we found that diversity needs to be reduced while syntheticity needs to be increased for scaling factor to go up, which can then improve overall accuracy. We include the constant values in Table \ref{['tab:param-estimates']}.
• Figure 4: This plot illustrates correlation between the accuracy and the scaling factor $Q$ across all model sizes, which shows that scaling up the value of $Q$ improves accuracy up to a point, where then the token number becomes dominant.
• Figure 5: This plot illustrates correlation between the predicted accuracy $G(N, D)$ and the true accuracy of RefinedWeb data. The Pearson correlation is +0.83.