Observations and Remedies for Large Language Model Bias in Self-Consuming Performative Loop

TL;DR

The paper investigates bias dynamics in Self-Consuming Performative Loop (SCPL) LLM training, focusing on two practical settings: incremental fine-tuning and retraining, under controlled performative feedback with group dynamics. It finds that SCPL amplifies preference bias while reducing disparate bias, with performative dynamics accelerating bias growth in some configurations; accumulation can slow bias progression and preserve some capabilities. To counter bias amplification, the authors propose a reward-based reweighting sampling strategy that guides data curation toward less biased samples while maintaining quality, showing strong mitigation performance across tasks. This work provides a principled framework and empirical evidence for safer self-improving systems and highlights directions for integrating more advanced alignment methods and multi-model feedback loops.

Abstract

The rapid advancement of large language models (LLMs) has led to growing interest in using synthetic data to train future models. However, this creates a self-consuming retraining loop, where models are trained on their own outputs and may cause performance drops and induce emerging biases. In real-world applications, previously deployed LLMs may influence the data they generate, leading to a dynamic system driven by user feedback. For example, if a model continues to underserve users from a group, less query data will be collected from this particular demographic of users. In this study, we introduce the concept of \textbf{S}elf-\textbf{C}onsuming \textbf{P}erformative \textbf{L}oop (\textbf{SCPL}) and investigate the role of synthetic data in shaping bias during these dynamic iterative training processes under controlled performative feedback. This controlled setting is motivated by the inaccessibility of real-world user preference data from dynamic production systems, and enables us to isolate and analyze feedback-driven bias evolution in a principled manner. We focus on two types of loops, including the typical retraining setting and the incremental fine-tuning setting, which is largely underexplored. Through experiments on three real-world tasks, we find that the performative loop increases preference bias and decreases disparate bias. We design a reward-based rejection sampling strategy to mitigate the bias, moving towards more trustworthy self-improving systems.

Figures (12)

• Figure 1: Illustration of the self-consuming performative loop for LLMs. Dynamic human feedback (e.g., an increase in queries from the blue group and a decrease from the pink group) influences both the generation of synthetic data and the subsequent training process.
• Figure 2: News Continuation
• Figure 3: Preference Dissection
• Figure 5: The disparate bias and mathematical problem solving ability on Math task using using Qwen2.5-Math-7B.
• Figure 6: The disparate bias and mathematical problem solving ability on Math task using Qwen2.5-Math-1.5B.