Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Alignment Game: A Theory of Long-Horizon Alignment Through Recursive Curation

Ali Falahati, Mohammad Mohammadi Amiri, Kate Larson, Lukasz Golab

TL;DR

The paper tackles long-horizon alignment for self-consuming generative models trained on their own outputs. It develops a formal two-agent BT-based curation model, treating theOwner and the Public as dynamic social choicers whose interactions over iterations shape the evolving distribution $p_t$. It reveals three convergence regimes—consensus trap under perfect alignment, compromise within shared optima under partial alignment, and owner-led refinement under disjoint alignment—plus a fundamental impossibility showing that diversity, symmetric influence, and initialization-independence cannot all be achieved simultaneously. Empirical studies in synthetic and text-based settings corroborate the theory, highlighting implications for designing transparent and contestable mechanisms to govern recursive alignment.

Abstract

In self-consuming generative models that train on their own outputs, alignment with user preferences becomes a recursive rather than one-time process. We provide the first formal foundation for analyzing the long-term effects of such recursive retraining on alignment. Under a two-stage curation mechanism based on the Bradley-Terry (BT) model, we model alignment as an interaction between two factions: the Model Owner, who filters which outputs should be learned by the model, and the Public User, who determines which outputs are ultimately shared and retained through interactions with the model. Our analysis reveals three structural convergence regimes depending on the degree of preference alignment: consensus collapse, compromise on shared optima, and asymmetric refinement. We prove a fundamental impossibility theorem: no recursive BT-based curation mechanism can simultaneously preserve diversity, ensure symmetric influence, and eliminate dependence on initialization. Framing the process as dynamic social choice, we show that alignment is not a static goal but an evolving equilibrium, shaped both by power asymmetries and path dependence.

The Alignment Game: A Theory of Long-Horizon Alignment Through Recursive Curation

TL;DR

The paper tackles long-horizon alignment for self-consuming generative models trained on their own outputs. It develops a formal two-agent BT-based curation model, treating theOwner and the Public as dynamic social choicers whose interactions over iterations shape the evolving distribution . It reveals three convergence regimes—consensus trap under perfect alignment, compromise within shared optima under partial alignment, and owner-led refinement under disjoint alignment—plus a fundamental impossibility showing that diversity, symmetric influence, and initialization-independence cannot all be achieved simultaneously. Empirical studies in synthetic and text-based settings corroborate the theory, highlighting implications for designing transparent and contestable mechanisms to govern recursive alignment.

Abstract

In self-consuming generative models that train on their own outputs, alignment with user preferences becomes a recursive rather than one-time process. We provide the first formal foundation for analyzing the long-term effects of such recursive retraining on alignment. Under a two-stage curation mechanism based on the Bradley-Terry (BT) model, we model alignment as an interaction between two factions: the Model Owner, who filters which outputs should be learned by the model, and the Public User, who determines which outputs are ultimately shared and retained through interactions with the model. Our analysis reveals three structural convergence regimes depending on the degree of preference alignment: consensus collapse, compromise on shared optima, and asymmetric refinement. We prove a fundamental impossibility theorem: no recursive BT-based curation mechanism can simultaneously preserve diversity, ensure symmetric influence, and eliminate dependence on initialization. Framing the process as dynamic social choice, we show that alignment is not a static goal but an evolving equilibrium, shaped both by power asymmetries and path dependence.

Paper Structure

This paper contains 20 sections, 7 theorems, 9 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Definition
  4. Model Components
  5. Alignment Regimes
  6. Assumptions
  7. Core Recursive Alignment Challenges
  8. Theoretical Results
  9. Perfect Alignment: The Consensus Trap
  10. Partial Alignment: The Compromise Equilibrium
  11. Disjoint Alignment: Asymmetric Power Dynamics
  12. The Fundamental Impossibility Result
  13. The Curation Mechanism as Social Choice
  14. Strategic Voting and Incentive Compatibility
  15. Fairness and Representation in Dynamic Voting
  16. ...and 5 more sections

Key Result

Theorem 1

Let the Owner and Public have perfectly aligned preferences: $A_O = A_P =: A_\star \neq \varnothing$ and $\{p_t\}_{t \geq 0}$ be the sequence of distributions generated by this mechanism. Then, the system reaches complete consensus: for any $\eta > 0$, there exist constants $C, c > 0$ such that $p_t

Figures (3)

  • Figure 1: KDE plots showing point distributions in three alignment scenarios: perfect alignment (same circles), partial alignment (overlapping circles), and disjoint alignment (non-overlapping circles). Red circles indicate owner preferred regions, blue dashed circles show public preferred regions.
  • Figure 2: (Top) Convergence of the mean distance to the owner and public centers over iterations for three alignment scenarios: perfect alignment, partial alignment, and disjoint alignment.(Bottom) The fraction of points within the owner and public preferred regions (“satisfaction”) as the alignment process progresses.
  • Figure 3: Word count evolution across three alignment scenarios: perfect alignment (agents target 4 words), partial alignment (owner targets 1-3 words, public targets 3-5 words), and disjoint alignment (owner targets 3-4 words, public targets 5-6 words).

Theorems & Definitions (11)

  • Definition 1
  • Definition 2
  • Theorem 1
  • Corollary 2: Mode Collapse for Unique Maximizers
  • Theorem 3
  • Theorem 4
  • Theorem 5
  • Remark 1
  • Definition 3
  • Theorem 6
  • ...and 1 more