Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Convergence and Stability Analysis of Self-Consuming Generative Models with Heterogeneous Human Curation

Hongru Zhao, Jinwen Fu, Tuan Pham

TL;DR

The paper analyzes convergence and stability of self-consuming generative models under heterogeneous human preferences, modeling the retraining loop as population-level updates with a finite or infinite candidate pool and an anchoring regularization parameter $\alpha$ that blends curated data with a reference distribution. It derives closed-form population updates across four regimes, proving KL-convergence in the pure synthetic case and contraction results (in TV or Hilbert projective metrics) when regularization is present, with an explicit fixed point structure in the infinite-pool regime. The main contributions are (i) a heterogeneous random-utility curation model, (ii) exact dynamics in four regimes, (iii) convergence guarantees and stability analyses under reward perturbations, and (iv) design guidance emphasizing the stabilizing role of reference data in curate-and-retrain loops. These results have practical implications for alignment pipelines (e.g., DPO/RLHF-style methods) by clarifying when and how anchor regularization ensures robust convergence and robustness to misspecified feedback.

Abstract

Self-consuming generative models have received significant attention over the last few years. In this paper, we study a self-consuming generative model with heterogeneous preferences that is a generalization of the model in Ferbach et al. (2024). The model is retrained round by round using real data and its previous-round synthetic outputs. The asymptotic behavior of the retraining dynamics is investigated across four regimes using different techniques including the nonlinear Perron--Frobenius theory. Our analyses improve upon that of Ferbach et al. (2024) and provide convergence results in settings where the well-known Banach contraction mapping arguments do not apply. Stability and non-stability results regarding the retraining dynamics are also given.

Convergence and Stability Analysis of Self-Consuming Generative Models with Heterogeneous Human Curation

TL;DR

The paper analyzes convergence and stability of self-consuming generative models under heterogeneous human preferences, modeling the retraining loop as population-level updates with a finite or infinite candidate pool and an anchoring regularization parameter that blends curated data with a reference distribution. It derives closed-form population updates across four regimes, proving KL-convergence in the pure synthetic case and contraction results (in TV or Hilbert projective metrics) when regularization is present, with an explicit fixed point structure in the infinite-pool regime. The main contributions are (i) a heterogeneous random-utility curation model, (ii) exact dynamics in four regimes, (iii) convergence guarantees and stability analyses under reward perturbations, and (iv) design guidance emphasizing the stabilizing role of reference data in curate-and-retrain loops. These results have practical implications for alignment pipelines (e.g., DPO/RLHF-style methods) by clarifying when and how anchor regularization ensures robust convergence and robustness to misspecified feedback.

Abstract

Self-consuming generative models have received significant attention over the last few years. In this paper, we study a self-consuming generative model with heterogeneous preferences that is a generalization of the model in Ferbach et al. (2024). The model is retrained round by round using real data and its previous-round synthetic outputs. The asymptotic behavior of the retraining dynamics is investigated across four regimes using different techniques including the nonlinear Perron--Frobenius theory. Our analyses improve upon that of Ferbach et al. (2024) and provide convergence results in settings where the well-known Banach contraction mapping arguments do not apply. Stability and non-stability results regarding the retraining dynamics are also given.

Paper Structure

This paper contains 29 sections, 33 theorems, 230 equations, 2 tables.

Table of Contents

  1. Introduction
  2. This paper
  3. Model Setup
  4. t-th round of curation-update
  5. Why we call it regularization.
  6. Notations
  7. Population-Level Retraining Dynamics with Curated Synthetic Data
  8. Retraining Dynamics Without Regularization
  9. Retraining Dynamics With Regularization
  10. Convergence Analysis
  11. Convergence in pure synthetic data retraining
  12. Convergence in retraining dynamics with regularization
  13. Convergence analysis in Regime (iii)
  14. Convergence analysis in Regime (iv)
  15. Stability under Reward Perturbations
  16. ...and 14 more sections

Key Result

Lemma 3.1

Let $\widehat{X}\sim \mathcal{PL}(X_{1:K},\varepsilon_{1:K})$ be the curated sample where $X_{1},\ldots,X_K\overset{\text{i.i.d.}}{\sim}p$. For any measurable function $f$ with $\mathbb{E}_{X\sim p}[|f(X)|]<\infty$, As a consequence, the density (w.r.t. $\pi$) of $\widehat{X}$ is $p H_p^K$.

Theorems & Definitions (79)

  • Remark 2.1: Heterogeneity and relation to DPO
  • Lemma 3.1
  • Theorem 3.2: Update Rule under Regime (i)
  • Lemma 3.3
  • Theorem 3.4: Update rule under Regime (ii)
  • Theorem 3.5: Update rule under Regime (iii)/(iv)
  • Lemma 4.1
  • Lemma 4.2
  • Theorem 4.3: Convergence under Regimes (i) and (ii)
  • Corollary 4.4
  • ...and 69 more