Synthetic continued pretraining

TL;DR

This work tackles the data-inefficiency of traditional pretraining when adapting large pretrained models to small, domain-specific corpora. It introduces synthetic continued pretraining (synthetic CPT) and a concrete instantiation, EntiGraph, which builds a knowledge graph from a small source corpus and generates diverse synthetic text describing relations among entities. Empirically, EntiGraph CPT exhibits a log-linear scaling of QA performance with synthetic tokens, and reaches about 80% of open-book knowledge accuracy relative to having the source documents available at inference time, with instruction-following capabilities and complementary gains to RAG in open-book settings. A simple mathematical model suggests EntiGraph rearranges existing knowledge into a structure that is more learnable, yielding a mixture-of-exponentials scaling that aligns with observed improvements, highlighting a general strategy for data-efficient, domain-specific pretraining through graph-based synthetic data generation.

Abstract

Pretraining on large-scale, unstructured internet text enables language models to acquire a significant amount of world knowledge. However, this knowledge acquisition is data-inefficient--to learn a given fact, models must be trained on hundreds to thousands of diverse representations of it. This poses a challenge when adapting a pretrained model to a small corpus of domain-specific documents, where each fact may appear rarely or only once. We propose to bridge this gap with synthetic continued pretraining: using the small domain-specific corpus to synthesize a large corpus more amenable to learning, and then performing continued pretraining on the synthesized corpus. We instantiate this proposal with EntiGraph, a synthetic data augmentation algorithm that extracts salient entities from the source documents and then generates diverse text by drawing connections between the sampled entities. Synthetic continued pretraining with EntiGraph enables a language model to answer questions and follow generic instructions related to the source documents without access to them. If, instead, the source documents are available at inference time, we show that the knowledge acquired through our approach compounds with retrieval-augmented generation. To better understand these results, we build a simple mathematical model of EntiGraph, and show how synthetic data augmentation can "rearrange" knowledge to enable more data-efficient learning.

Figures (8)

• Figure 1: Synthetic continued pretraining (synthetic CPT) converts a small source corpus into a large synthetic corpus that is amenable to learning via standard continued pretraining. We instantiate synthetic CPT using a synthetic data augmentation algorithm called EntiGraph, which forms a knowledge graph over entities extracted from documents, and then prompts an LM to synthesize a text-based representation of the graph.
• Figure 2: Accuracy on the QuALITY question set ${\mathcal{Q}}_{\text{test}}$ ($y$-axis) as a function of the synthetic token count ($x$-axis). The accuracy of synthetic continued pretraining using the EntiGraph data augmentation algorithm (EntiGraph CPT) scales log-linearly up to 455M tokens.
• Figure 3: Closed-book summarization: number of false claims ($y$-axis) versus number of salient claims ($x$-axis) normalized by the human summary.
• Figure 4: A mixture-of-exponential functional form \ref{['eqn:moe']} closely fits the scaling trend of EntiGraph CPT with respect to synthetic token count.
• Figure 5: Accuracy $\mathsf{Acc}({\boldsymbol M}_t)$ with respect to time $t$, for $V = 100$ and $p=0.03$. The mixture-of-exponential functional form in \ref{['eqn:moe']} leads to three distinct regimes.

Theorems & Definitions (7)

• Definition 1
• Theorem 1
• proof : Proof of Theorem \ref{['thm:toy']}
• Lemma F.1: Lemma 1 and Corollary 1 in karp1990transitive
• Lemma F.2: Theorem 3 in karp1990transitive and Theorem 2.4.1 in durrett2010random
• Lemma F.3
• proof : Proof of Lemma \ref{['lem:shape']}