Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Self-Consuming Generative Models with Adversarially Curated Data

Xiukun Wei, Xueru Zhang

TL;DR

This work analyzes how self-consuming generative models evolve when trained on data curated adversarially by users. It develops a theoretical framework showing conditions under which the training loop remains robust to noisy curation and delineates scenarios where misalignment with genuine user preferences emerges due to covariances between reward signals. The authors propose gradient-based and heuristic attack algorithms to degrade alignment, and validate them with experiments on synthetic Gaussian data and real-world CIFAR datasets, demonstrating the practicality and effectiveness of such attacks. The results highlight vulnerabilities in self-consuming retraining and motivate future defenses that can jointly account for adversarial data curation and dynamics of iterative model updates.

Abstract

Recent advances in generative models have made it increasingly difficult to distinguish real data from model-generated synthetic data. Using synthetic data for successive training of future model generations creates "self-consuming loops", which may lead to model collapse or training instability. Furthermore, synthetic data is often subject to human feedback and curated by users based on their preferences. Ferbach et al. (2024) recently showed that when data is curated according to user preferences, the self-consuming retraining loop drives the model to converge toward a distribution that optimizes those preferences. However, in practice, data curation is often noisy or adversarially manipulated. For example, competing platforms may recruit malicious users to adversarially curate data and disrupt rival models. In this paper, we study how generative models evolve under self-consuming retraining loops with noisy and adversarially curated data. We theoretically analyze the impact of such noisy data curation on generative models and identify conditions for the robustness of the retraining process. Building on this analysis, we design attack algorithms for competitive adversarial scenarios, where a platform with a limited budget employs malicious users to misalign a rival's model from actual user preferences. Experiments on both synthetic and real-world datasets demonstrate the effectiveness of the proposed algorithms.

Self-Consuming Generative Models with Adversarially Curated Data

TL;DR

This work analyzes how self-consuming generative models evolve when trained on data curated adversarially by users. It develops a theoretical framework showing conditions under which the training loop remains robust to noisy curation and delineates scenarios where misalignment with genuine user preferences emerges due to covariances between reward signals. The authors propose gradient-based and heuristic attack algorithms to degrade alignment, and validate them with experiments on synthetic Gaussian data and real-world CIFAR datasets, demonstrating the practicality and effectiveness of such attacks. The results highlight vulnerabilities in self-consuming retraining and motivate future defenses that can jointly account for adversarial data curation and dynamics of iterative model updates.

Abstract

Recent advances in generative models have made it increasingly difficult to distinguish real data from model-generated synthetic data. Using synthetic data for successive training of future model generations creates "self-consuming loops", which may lead to model collapse or training instability. Furthermore, synthetic data is often subject to human feedback and curated by users based on their preferences. Ferbach et al. (2024) recently showed that when data is curated according to user preferences, the self-consuming retraining loop drives the model to converge toward a distribution that optimizes those preferences. However, in practice, data curation is often noisy or adversarially manipulated. For example, competing platforms may recruit malicious users to adversarially curate data and disrupt rival models. In this paper, we study how generative models evolve under self-consuming retraining loops with noisy and adversarially curated data. We theoretically analyze the impact of such noisy data curation on generative models and identify conditions for the robustness of the retraining process. Building on this analysis, we design attack algorithms for competitive adversarial scenarios, where a platform with a limited budget employs malicious users to misalign a rival's model from actual user preferences. Experiments on both synthetic and real-world datasets demonstrate the effectiveness of the proposed algorithms.
Paper Structure (22 sections, 7 theorems, 52 equations, 10 figures, 1 table, 2 algorithms)

This paper contains 22 sections, 7 theorems, 52 equations, 10 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Impact of adversarially curated data
  4. Attack algorithms
  5. Gradient-based methods
  6. Heuristic methods
  7. Experiments
  8. Evolution of model under adversarial data curation
  9. Effectiveness of attack methods
  10. Discussion
  11. Conclusion
  12. Related work
  13. Self-consuming generative models
  14. Data poisoning attack
  15. Proofs
  16. ...and 7 more sections

Key Result

Lemma 3.1

Consider the asymptotic case where the number of samples users select from satisfies $K \to \infty$. Suppose $\mathbb{E}_{x \sim p_{t}}[e^{r(x)}] < \infty$ and $p_{t+1}$ follows Eq. SyInter. Then, we have

Figures (10)

  • Figure 1: An example of adversarial data curation in a competitive setting: the adversarial platform and the target platform serve the same population of users. The adversarial platform can access the data generated by the target platform and obtain real user preferences through its own preference collection mechanisms. Using the attack algorithm, the adversarial platform employs malicious users to adversarially curate data on the target platform, preventing its model from aligning with genuine user preferences.
  • Figure 2: The proportion of each class generated by the self-consumption model retrained with different data curation methods on CIFAR-10: benign curation based on actual user preferences (left), adversarial curation using the proposed gradient-based attack algorithm (middle), and adversarial curation via a random attack (right). The results show that the proposed gradient-based attacks are the most effective in deviating the model from user preferences.
  • Figure 3: The proportion of each ten classes generated by the self-consumption model retrained with different data curation methods on CIFAR-100: benign curation based on actual user preferences (left), adversarial curation using the proposed gradient-based attack algorithm (middle). And empirical estimate of $\mathbb{E}_{p_t}[r(x)]$ from samples generated by the model over iterations (right).
  • Figure 4: Empirical estimate of $\mathbb{E}_{p_t}[r(x)]$ from model-generated samples over iterations on CIFAR-10: it increases (resp. decreases) over iterations under benign (resp. adversarial) data curation.
  • Figure 5: The proportion of each class generated by the self-consumption model retrained with a mix of adversarially curated synthetic and real CIFAR-10 data. It shows that adding real data only helps the model align with the real data distribution $p_{\text{data}}$ but does not defend against adversarial data curation.
  • ...and 5 more figures

Theorems & Definitions (12)

  • Lemma 3.1
  • Lemma 3.3
  • Lemma 3.4
  • Lemma 2.1
  • proof
  • Lemma 2.2
  • proof
  • Lemma 2.3
  • proof
  • proof
  • ...and 2 more