How Bad is Training on Synthetic Data? A Statistical Analysis of Language Model Collapse

TL;DR

The paper investigates model collapse in recursive language-model training where synthetic data are generated from prior models. It presents a simple statistical framework for next-token prediction and proves that training exclusively on synthetic data leads to almost-sure total collapse, with $S_m = 1 - (1 - 1/n)^m (1 - S_0)$ describing the variance trajectory and absorbing Dirac-mass convergence. When real data are mixed in (Partially Synthetic setting), the authors derive a closed-form evolution and concentration-based bounds that quantify how much synthetic data can be injected before collapse becomes unavoidable, linking the bound to $\alpha = n/(N+n)$ and $\beta$ as defined, and they bound $\mathbb{E}\|{\mathbf{p}}^{(m)} - {\mathbf{p}}^{(1)}\|_1$. Empirical validations with GPT2-style models on real text and synthetic experiments corroborate the theory, showing collapse in the fully synthetic regime and stability when real data are injected, thereby providing practical guidance on data mixtures to mitigate distribution drift in future generations of language models.

Abstract

The phenomenon of model collapse, introduced in (Shumailov et al., 2023), refers to the deterioration in performance that occurs when new models are trained on synthetic data generated from previously trained models. This recursive training loop makes the tails of the original distribution disappear, thereby making future-generation models forget about the initial (real) distribution. With the aim of rigorously understanding model collapse in language models, we consider in this paper a statistical model that allows us to characterize the impact of various recursive training scenarios. Specifically, we demonstrate that model collapse cannot be avoided when training solely on synthetic data. However, when mixing both real and synthetic data, we provide an estimate of a maximal amount of synthetic data below which model collapse can eventually be avoided. Our theoretical conclusions are further supported by empirical validations.

Figures (6)

• Figure 1: Evolution of ${\bm{p}}^{(m)}$ in the Fully Synthetic setting for vocabulary size $s=3$, context length $\ell=4$, total contexts $c=s^\ell = 81$ and sample size $n=1000$. The initial distribution ${\bm{p}}^{(0)}$ is some random distribution over tokens. The trained conditional distributions converge towards Dirac measures over generations illustrating total collapse in Definition \ref{['def_total_collapse']}.
• Figure 2: Fully Synthetic case for different initial distribution ${\bm{p}}^{(0)}$ .Total collapse time is plotted as a function of the initial distribution ${\bm{p}}$ and the sample size $n$. (Top) The initial distribution ${\bm{p}}$ with different values of $S_0$ and support size $\tilde{s}$. The $x$-axis represents tokens $i\in \{1,2,\ldots, 600\}$ while the $y$-axis represents the probabilities in log scale. (Bottom) Each cross represents the average total collapse time over $100$ runs for a particular sample sizes $n\in \{10, 50, 100, 150, \ldots, 400\}$. The red dashed line depicts the lower bound on $\mathbb{E} T$ given by (\ref{['eq:case1:expectation']}).
• Figure 3: Partially Synthetic case with different sample sizes $n$.A hundred experiments were run for $50$ generations for $N=100$ and different values of $n$. Each yellow line represents the evolution of $\sigma_m$ (top row) or $\|{\bm{p}}^{(m)}-{\bm{p}}^{(1)}\|_1$ (bottom row) in one experiment, with the red line being the empirical mean across $100$ runs. The blue dashed lines plot the formula for $S_m$ given by (\ref{['eq:case2-sm']}). The initial distribution ${\bm{p}}$ satisfies $s=600$, $\tilde{s}=52$, and $S_0=0.1$.
• Figure 4: Partially Synthetic case with different initial distributions ${\bm{p}}$.A hundred experiments were run for $50$ generations for $n=10$, $N=100$ and different initial distributions ${\bm{p}}$ shown in the top row. Notice that as $S_0$ increases, the deviation $\|p^{(m)}-p^{(1)}\|_1$ decreases, as suggest by the inequality (\ref{['ineq:bound-1-norm']}).
• Figure 5: Experiments with a GPT2-type generative model. The top left plot depicts the deviation $\Vert {\bm{p}}^{(m)} - {\bm{p}}^{(1)} \Vert_1$ varying the generation model $m$ for synthetic data and a mixture of real and synthetic data. The three other plots show the behavior of the validation loss over generations. Essentially, training solely on synthetic data causes model collapse and affects the usual scaling laws dohmatob2024model.

Theorems & Definitions (13)

• Definition 1: Total Collapse
• Theorem 1: Control on Total Collapse
• Proposition 1
• Theorem 2: Model Variation
• Theorem 3: Model Deviation
• Lemma 1
• proof
• proof : Proof of Theorem \ref{['theorem:case-1']}
• proof : Proof of Proposition \ref{['proposition:case-1-limit']}
• proof : Proof of Theorem \ref{['theorem:case-2-variance']}