Scaling Laws of Synthetic Data for Language Models

TL;DR

The paper addresses data scarcity for scaling language models by proposing SynthLLM, a scalable framework that converts vast pre-training corpora into high-quality synthetic data through domain filtering, concept-grounded question generation, and answer generation. It demonstrates that synthetic data obeys a rectified scaling law, with diminishing returns around 300B tokens and model-size dependent data needs, enabling accurate performance forecasting as data scales. Empirical results in math reasoning—and preliminary coding-domain experiments—show SynthLLM outperforms existing synthetic-generation baselines and augmentation methods, with Level-2 and Level-3 methods delivering the strongest gains due to cross-document grounding and concept decomposition. These findings support synthetic data as a practical, scalable path to continued improvements in LLM capabilities, potentially reducing reliance on organic corpora while extending performance across domains.

Abstract

Large language models (LLMs) achieve strong performance across diverse tasks, largely driven by high-quality web data used in pre-training. However, recent studies indicate this data source is rapidly depleting. Synthetic data emerges as a promising alternative, but it remains unclear whether synthetic datasets exhibit predictable scalability comparable to raw pre-training data. In this work, we systematically investigate the scaling laws of synthetic data by introducing SynthLLM, a scalable framework that transforms pre-training corpora into diverse, high-quality synthetic datasets. Our approach achieves this by automatically extracting and recombining high-level concepts across multiple documents using a graph algorithm. Key findings from our extensive mathematical experiments on SynthLLM include: (1) SynthLLM generates synthetic data that reliably adheres to the rectified scaling law across various model sizes; (2) Performance improvements plateau near 300B tokens; and (3) Larger models approach optimal performance with fewer training tokens. For instance, an 8B model peaks at 1T tokens, while a 3B model requires 4T. Moreover, comparisons with existing synthetic data generation and augmentation methods demonstrate that SynthLLM achieves superior performance and scalability. Our findings highlight synthetic data as a scalable and reliable alternative to organic pre-training corpora, offering a viable path toward continued improvement in model performance.

Figures (12)

• Figure 1: Scaling laws across different model sizes. The x-axis denotes the number of training tokens, while the y-axis shows error rates on the MATH benchmark. Green points indicate data sizes used for fitting the scaling curves, and red points assess the predictive accuracy of these fitted curves.
• Figure 2: Overview of reference documents filtering.
• Figure 3: Illustrations of our Document-Grounded Question Generation methods.
• Figure 4: Histogram of question similarity within the same document. The x-axis represents the cosine similarity between question pairs from the same document, while the y-axis denotes the frequency of each bin.
• Figure 5: (a): The performances of Level-2, Level-3, and other augmentation methods on MATH benchmark. (b): The average performances across various benchmarks. The x axis denotes the sample number. The y axis represents the accuracy (%).