Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Preventing Model Collapse via Contraction-Conditioned Neural Filters

Zongjian Han, Yiran Liang, Ruiwen Wang, Yiwei Luo, Yilin Huang, Xiaotong Song, Dongqing Wei

TL;DR

The paper tackles model-collapse risk in recursive training with self-generated synthetic data by reframing estimation error as a nonlinear stochastic process and enforcing a contraction condition via a learned neural filter. It introduces a contraction-operator framework, incorporating a Lyapunov-based contraction trigger and a data-filter network that selects informative samples to ensure convergence even with constant sample sizes. The authors prove probabilistic convergence guarantees (Theorems 1 and 3) and demonstrate empirically that the contraction-conditioned filter stabilizes parameter estimates and prevents collapse, outperforming the prior superlinear-sample-growth approach. Practical implications are highlighted for large language models, data augmentation, privacy-preserving synthesis, and continual learning, where stable long-term training is essential.

Abstract

This paper presents a neural network filter method based on contraction operators to address model collapse in recursive training of generative models. Unlike \cite{xu2024probabilistic}, which requires superlinear sample growth ($O(t^{1+s})$), our approach completely eliminates the dependence on increasing sample sizes within an unbiased estimation framework by designing a neural filter that learns to satisfy contraction conditions. We develop specialized neural network architectures and loss functions that enable the filter to actively learn contraction conditions satisfying Assumption 2.3 in exponential family distributions, thereby ensuring practical application of our theoretical results. Theoretical analysis demonstrates that when the learned contraction conditions are satisfied, estimation errors converge probabilistically even with constant sample sizes, i.e., $\limsup_{t\to\infty}\mathbb{P}(\|\mathbf{e}_t\|>δ)=0$ for any $δ>0$. Experimental results show that our neural network filter effectively learns contraction conditions and prevents model collapse under fixed sample size settings, providing an end-to-end solution for practical applications.

Preventing Model Collapse via Contraction-Conditioned Neural Filters

TL;DR

The paper tackles model-collapse risk in recursive training with self-generated synthetic data by reframing estimation error as a nonlinear stochastic process and enforcing a contraction condition via a learned neural filter. It introduces a contraction-operator framework, incorporating a Lyapunov-based contraction trigger and a data-filter network that selects informative samples to ensure convergence even with constant sample sizes. The authors prove probabilistic convergence guarantees (Theorems 1 and 3) and demonstrate empirically that the contraction-conditioned filter stabilizes parameter estimates and prevents collapse, outperforming the prior superlinear-sample-growth approach. Practical implications are highlighted for large language models, data augmentation, privacy-preserving synthesis, and continual learning, where stable long-term training is essential.

Abstract

This paper presents a neural network filter method based on contraction operators to address model collapse in recursive training of generative models. Unlike \cite{xu2024probabilistic}, which requires superlinear sample growth (), our approach completely eliminates the dependence on increasing sample sizes within an unbiased estimation framework by designing a neural filter that learns to satisfy contraction conditions. We develop specialized neural network architectures and loss functions that enable the filter to actively learn contraction conditions satisfying Assumption 2.3 in exponential family distributions, thereby ensuring practical application of our theoretical results. Theoretical analysis demonstrates that when the learned contraction conditions are satisfied, estimation errors converge probabilistically even with constant sample sizes, i.e., for any . Experimental results show that our neural network filter effectively learns contraction conditions and prevents model collapse under fixed sample size settings, providing an end-to-end solution for practical applications.

Paper Structure

This paper contains 34 sections, 8 theorems, 58 equations, 7 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. The Challenge of Model Collapse in the Age of Synthetic Data
  3. Foundations: Models, Parameters, and Recursive Training
  4. A Probabilistic Foundation: Random Walks in Parameter Space
  5. The Prevailing Solution: A Theoretically Sound but Practically Limiting Strategy
  6. Our Approach: Steering the Random Walk with Learned Contraction Filters
  7. Summary of Contributions
  8. Mathematical Modeling and Deduction
  9. Mathematical Modeling
  10. Main Theoretical Results
  11. The Estimation of Convergence Rate
  12. The Combination with the Work of Predecessors
  13. Neural Architecture Design
  14. Problem Statement
  15. Neural Architecture Design
  16. ...and 19 more sections

Key Result

Theorem 2.7

If a stochastic difference system eq1 satisfies Assumption as1, as2 and as3, then for each $\delta>0$$\limsup\limits_{t \to \infty} \mathbb{P}(\|\mathbf{e}_t\| > \delta)=0$ , which means the model will not collapse.

Figures (7)

  • Figure 1: A General Framework for Recursive Training with Fully Synthetic Data.xu2024probabilistic
  • Figure 2: This image explains the principle of model collapse from the perspective of random walks. xu2024probabilistic
  • Figure 3: This image explains the principle of model collapse from the perspective of random walks.xu2024probabilistic
  • Figure 4: This figure explains the working mechanism of the filter: by continuously pulling the parameters back to the origin, it prevents the model from collapsing.
  • Figure 5: This figure show the relationship between the contraction strength and the vector distance from the origin which is the parameter error.
  • ...and 2 more figures

Theorems & Definitions (16)

  • Remark 2.2
  • Remark 2.4
  • Remark 2.6
  • Theorem 2.7
  • Theorem 2.10: Convergence Rates for Recurrence Inequality
  • Theorem 2.12
  • Remark 2.13
  • Lemma 1.1
  • proof
  • Lemma 1.2
  • ...and 6 more