Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

What happens when generative AI models train recursively on each others' outputs?

Hung Anh Vu, Galen Reeves, Emily Wenger

TL;DR

The paper addresses how generative AI models might be trained on outputs from other models, proposing a formal interactive training framework parameterized by $\alpha$ and $\beta$. It derives exact mean and covariance recursions for a linear-Gaussian surrogate (via matrices $\bm{M}_t$, $\bm{C}_t$, $\bm{P}_t$, $\bm{Q}_t$) and identifies conditions for unbiasedness and convergence (notably $\rho(\bm{Q})<1$ and cases where $S$ is proportional to $\tilde{S}$). Empirically, two-model experiments with synthetic data corroborate the theory: mixing real and synthetic data can improve cross-task transfer but tends to homogenize representations, with optimal performance near $\alpha=\beta=0.5$. The findings have implications for data curation and model-update strategies in multi-model ecosystems, highlighting both potential benefits and risks of data-mediated interactions in future AI systems.

Abstract

The internet serves as a common source of training data for generative AI (genAI) models but is increasingly populated with AI-generated content. This duality raises the possibility that future genAI models may be trained on other models' generated outputs. Prior work has studied consequences of models training on their own generated outputs, but limited work has considered what happens if models ingest content produced by other models. Given society's increasing dependence on genAI tools, understanding such data-mediated model interactions is critical. This work provides empirical evidence for how data-mediated interactions might unfold in practice, develops a theoretical model for this interactive training process, and experimentally validates the theory. We find that data-mediated interactions can benefit models by exposing them to novel concepts perhaps missed in original training data, but also can homogenize their performance on shared tasks.

What happens when generative AI models train recursively on each others' outputs?

TL;DR

The paper addresses how generative AI models might be trained on outputs from other models, proposing a formal interactive training framework parameterized by and . It derives exact mean and covariance recursions for a linear-Gaussian surrogate (via matrices , , , ) and identifies conditions for unbiasedness and convergence (notably and cases where is proportional to ). Empirically, two-model experiments with synthetic data corroborate the theory: mixing real and synthetic data can improve cross-task transfer but tends to homogenize representations, with optimal performance near . The findings have implications for data curation and model-update strategies in multi-model ecosystems, highlighting both potential benefits and risks of data-mediated interactions in future AI systems.

Abstract

The internet serves as a common source of training data for generative AI (genAI) models but is increasingly populated with AI-generated content. This duality raises the possibility that future genAI models may be trained on other models' generated outputs. Prior work has studied consequences of models training on their own generated outputs, but limited work has considered what happens if models ingest content produced by other models. Given society's increasing dependence on genAI tools, understanding such data-mediated model interactions is critical. This work provides empirical evidence for how data-mediated interactions might unfold in practice, develops a theoretical model for this interactive training process, and experimentally validates the theory. We find that data-mediated interactions can benefit models by exposing them to novel concepts perhaps missed in original training data, but also can homogenize their performance on shared tasks.

Paper Structure

This paper contains 17 sections, 4 theorems, 34 equations, 7 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. How Today's Large-Scale Generative AI Models Are Trained
  4. Formalizing An Iterative, Interactive Model Training Pipeline
  5. Theory
  6. Experimental Evaluation
  7. Discussion
  8. Exact training data statements from LLM papers
  9. Counterarguments
  10. Additional Results
  11. Code for experiments
  12. Proofs for results in Section \ref{['sec:theory']}
  13. Linear dynamical system
  14. Proof of Theorem \ref{['thm:MtCt']}
  15. Proof of Theorem \ref{['thm:bias_mse']}
  16. ...and 2 more sections

Key Result

Theorem 1

Conditional on the initial data $D_0 \coloneqq (\tilde{D}_1, \dots, \tilde{D}_K, D_*)$, the estimates $\hat{\bm{\theta}}_t = \mathop{\mathrm{\mathsf{vec}}}\nolimits( \hat{\theta}_{t1}, \dots, \hat{\theta}_{tK})$ are Gaussian with mean and variance where the matrices $\bm{M}_t$ and $\bm{C}_t$ are defined recursively with $\bm{M}_0 = \bm{P}_0$ and $\bm{C}_0 = \bm{0}_{Kd \times Kd}$ and

Figures (7)

  • Figure 1: Our dataset update scheme, parameterized by $\alpha$ and $\beta$.This paradigm best aligns with evidence from the literature given in §\ref{['sec:realities']} and strongly indicates that interactions between models, facilitating by training on others' generated data, are an important consideration for empirical and theoretical work on model evolution.
  • Figure 2: Predicted relative efficiency across $\alpha, \beta$ values for a $K=4$ model system (dimension $15$ and rank $5$).Curves show ratio of MSE of the minimum variance unbiased estimator to asymptotic MSE for a given $\alpha, \beta$ setting obtained from Theorem \ref{['thm:asymp_var']} ($1$ is optimal). Setting $\alpha=\beta=0.5$ produces best results across models.
  • Figure 3: PCA of embeddings of outputs produced by models $\theta_{t1}$ and $\theta_{t2}$ on datasets $\tilde{\mathcal{D}}_1$, $\tilde{\mathcal{D}}_2$, and $\tilde{\mathcal{D}}_*$. Results for $K=2$ Llama models trained with $\beta=0.5$ and varying $\alpha$. Leftmost plot shows embeddings at generation $t=0$, which are identical for all $\alpha$; while right plots show embeddings at $t=15$ with different $\alpha$.
  • Figure 4: Predicted behavior over time for a $K=2$ model system with varying $\alpha$, $\beta$.We use equations for MSE from theorem \ref{['thm:bias_mse']} and run simulations with $K=2$, dimension 50, rank 15.
  • Figure 5: Actual behavior over time for interactions between OPT models ($K=2$) with varying $\alpha,\beta$.
  • ...and 2 more figures

Theorems & Definitions (4)

  • Theorem 1
  • Theorem 2
  • Lemma 1
  • Theorem 3