Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Your Self-Play Algorithm is Secretly an Adversarial Imitator: Understanding LLM Self-Play through the Lens of Imitation Learning

Shangzhe Li, Xuchao Zhang, Chetan Bansal, Weitong Zhang

TL;DR

This work reframes self-play finetuning for large language models as an adversarial imitation learning problem, introducing a min–max game between a policy (the model) and a reward learner (parameterized by the model and past snapshots). It develops both a single-stage and a two-stage, $\chi^2$-divergence–based framework, establishing convergence guarantees and showing that a bounded reward via a mixed $\chi^2$ regularizer yields improved stability. The proposed SPIF algorithm turns the two-stage optimization into a practical least-squares objective with data mixing, enabling robust training and stable gradients while achieving consistent improvements over SPIN and SFT across several benchmarks. Theoretical analysis connects self-play with general preference alignment, highlighting the role of distribution matching and regularization in principled self-improvement. Empirically, SPIF demonstrates stronger instruction-following performance and more stable dynamics, underscoring the practical impact of a principled AIL interpretation for self-play finetuning.

Abstract

Self-play post-training methods has emerged as an effective approach for finetuning large language models and turn the weak language model into strong language model without preference data. However, the theoretical foundations for self-play finetuning remain underexplored. In this work, we tackle this by connecting self-play finetuning with adversarial imitation learning by formulating finetuning procedure as a min-max game between the model and a regularized implicit reward player parameterized by the model itself. This perspective unifies self-play imitation and general preference alignment within a common framework. Under this formulation, we present a game-theoretic analysis showing that the self-play finetuning will converge to it's equilibrium. Guided by this theoretical formulation, we propose a new self-play imitation finetuning algorithm based on the $χ^2$-divergence variational objective with bounded rewards and improved stability. Experiments on various of language model finetuning tasks demonstrate consistent improvements over existing self-play methods and validate our theoretical insights.

Your Self-Play Algorithm is Secretly an Adversarial Imitator: Understanding LLM Self-Play through the Lens of Imitation Learning

TL;DR

This work reframes self-play finetuning for large language models as an adversarial imitation learning problem, introducing a min–max game between a policy (the model) and a reward learner (parameterized by the model and past snapshots). It develops both a single-stage and a two-stage, -divergence–based framework, establishing convergence guarantees and showing that a bounded reward via a mixed regularizer yields improved stability. The proposed SPIF algorithm turns the two-stage optimization into a practical least-squares objective with data mixing, enabling robust training and stable gradients while achieving consistent improvements over SPIN and SFT across several benchmarks. Theoretical analysis connects self-play with general preference alignment, highlighting the role of distribution matching and regularization in principled self-improvement. Empirically, SPIF demonstrates stronger instruction-following performance and more stable dynamics, underscoring the practical impact of a principled AIL interpretation for self-play finetuning.

Abstract

Self-play post-training methods has emerged as an effective approach for finetuning large language models and turn the weak language model into strong language model without preference data. However, the theoretical foundations for self-play finetuning remain underexplored. In this work, we tackle this by connecting self-play finetuning with adversarial imitation learning by formulating finetuning procedure as a min-max game between the model and a regularized implicit reward player parameterized by the model itself. This perspective unifies self-play imitation and general preference alignment within a common framework. Under this formulation, we present a game-theoretic analysis showing that the self-play finetuning will converge to it's equilibrium. Guided by this theoretical formulation, we propose a new self-play imitation finetuning algorithm based on the -divergence variational objective with bounded rewards and improved stability. Experiments on various of language model finetuning tasks demonstrate consistent improvements over existing self-play methods and validate our theoretical insights.
Paper Structure (35 sections, 14 theorems, 100 equations, 3 figures, 5 tables, 2 algorithms)

This paper contains 35 sections, 14 theorems, 100 equations, 3 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. Preliminaries
  4. An Adversarial Imitation Learning View
  5. A Single-Stage Formulation
  6. A Two-Stage Iterative Alternative
  7. A Game-Theoretic Analysis
  8. $\chi^2$ Self-Play Imitation Finetuning
  9. Empirical Results
  10. Discussion
  11. Adversarial Imitation Learning.
  12. Self-Play Imitation Finetuning
  13. Self-Play for General Preference Alignment
  14. Conclusion
  15. Proof of Theorem \ref{['thm:upper-dual']}
  16. ...and 20 more sections

Key Result

Proposition 4.1

The mixed Pearson $\chi^2$ divergence between $\pi^\star$ and the mixture distribution $\tfrac{\pi+\pi^\star}{2}$ induced by the convex regularizer $\psi(r)=\tfrac{c}{2} \cdot\left({\mathbb{E}}_{\pi^\star}[(r(x,y))^2] + {\mathbb{E}}_{\pi}[(r(x,y))^2]\right)$ is bounded by: Furthermore, the optimal reward for solving the variational form of this Pearson $\chi^2$ divergence: is bounded within the

Figures (3)

  • Figure 1: Reward Dynamics Analysis. We plot the reward curves (log-scaled) during training for our approach, SPIF with $\chi^2$ regularization, and for SPIN chen2024self. The results show that our method produces rewards with substantially smaller magnitude, which leads to more stable learning dynamics and is consistent with our theoretical analysis predicting a tighter duality gap.
  • Figure 2: Gradient Norm Analysis. We plot the gradient norms (log-scaled) during training for our approach, SPIF with $\chi^2$ regularization, and for SPIN chen2024self. The results show that our method maintains significantly more stable gradient norms, indicating improved training stability compared to SPIN.
  • Figure 3: Ablation on Hyperparameter $c$. We evaluate the impact of the hyperparameter $c$ by setting $c \in \{0.125, 0.5, 2\}$ and examining its effect on the self-play performance of our method. Performance is measured as the mean score across the four benchmarks used in the main experiments. We observe that both overly small and overly large values of $c$ lead to performance degradation, highlighting the importance of an appropriate balance in reward scaling.

Theorems & Definitions (22)

  • Proposition 4.1: Contextual bandit version of Proposition A.2 and A.3, al2023ls
  • Proposition 4.2
  • Definition 4.3
  • Theorem 4.4
  • Remark 4.5
  • Remark 4.6
  • Remark 4.7
  • Proposition 5.1
  • Proposition 5.2
  • Remark 5.3
  • ...and 12 more