Escaping Model Collapse via Synthetic Data Verification: Near-term Improvements and Long-term Convergence

TL;DR

This work analyzes how verifier-based synthetic retraining can mitigate model collapse when training on self-generated data by introducing a binary verifier feedback signal. By formalizing the process in a linear regression setting and validating with VAEs on MNIST, it derives a bias–variance decomposition showing conditions under which one-step improvements occur and proves that iterative retraining converges toward the verifier's knowledge center $\theta_c$ when synthetic data grow without bound; the long-term outcome is driven by the verifier's bias. Empirically, the theory aligns with simulations and CVAE experiments: early gains materialize from variance reduction via filtering, but sustained improvement requires a highly reliable verifier, as bias can cause convergence to a biased fixed point. Overall, the study provides principled guidance on when verifier-based filtering helps and highlights the practical trade-offs for real-world data-limited regimes. The findings have implications for deploying synthetic data pipelines in vision and representation learning, informing verifier design and data sourcing choices to balance short-term gains against long-run convergence to external knowledge.

Abstract

Synthetic data has been increasingly used to train frontier generative models. However, recent study raises key concerns that iteratively retraining a generative model on its self-generated synthetic data may keep deteriorating model performance, a phenomenon often coined model collapse. In this paper, we investigate ways to modify this synthetic retraining process to avoid model collapse, and even possibly help reverse the trend from collapse to improvement. Our key finding is that by injecting information through an external synthetic data verifier, whether a human or a better model, synthetic retraining will not cause model collapse. To develop principled understandings of the above insight, we situate our analysis in the foundational linear regression setting, showing that iterative retraining with verified synthetic data can yield near-term improvements but ultimately drives the parameter estimate to the verifier's "knowledge center" in the long run. Our theory hence predicts that, unless the verifier is perfectly reliable, the early gains will plateau and may even reverse. Indeed, these theoretical insights are further confirmed by our experiments on both linear regression as well as Variational Autoencoders (VAEs) trained on MNIST data.

Figures (6)

• Figure 1: Iterative Retraining of a Variational Autoencoder (VAE) on MNIST.Left: Original MNIST images used as the real training data. Middle: Images generated by the initial VAE trained on only 500 real images. Right: Images after 40 rounds of synthetic retraining. The top branch (green) retrains with filtered synthetic data by a verifier, yielding much clearer and more realistic digits; the bottom branch (red) retrains without a verifier, leading to severe degradation and mode collapse. For reference, the final column shows samples from a VAE trained on all 60K real images, representing the upper bound of achievable quality.
• Figure 2: Generate-Verify-Retrain pipeline corresponding to Scheme \ref{['scheme:linear_retraining']}.
• Figure 3: Error changes of the one-step retraining estimator $\hat{\theta}^1$ versus estimator $\hat{\theta}^0$ only using original real data, measured by $\log(\frac{||\hat{\theta}^0 - \theta^\star||}{||\hat{\theta}^1 - \theta^\star||})$: theory's prediction (left) and empirical comparisons (right).
• Figure 4: Iterative synthetic retraining with and without bias.
• Figure 5: FID results across retraining rounds. (a) Effect of filtering and retained sample size. (b) Effect of verifier quality, varied by training data size. Together, the plots highlight how both sample selection and verifier selectivity shape generative performance.

Theorems & Definitions (9)

• Theorem 3.1
• Remark 1
• Theorem 4.1
• Theorem A.1
• proof : Proof of Theorem \ref{['thm:gaussian:main']}
• Theorem A.2
• proof : Proof of Theorem \ref{['thm:gaussian:limit']}
• proof : Proof of Theorem \ref{['thm:linear:one_step']}
• proof : Proof of Theorem \ref{['thm:linear:long_term']}