Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Self-Generative Adversarial Fine-Tuning for Large Language Models

Shiguang Wu, Yaqing Wang, Quanming Yao

TL;DR

SGALM reframes LLM alignment as a self-contained adversarial game in which a single model alternately acts as generator and discriminator via in-context learning and its own output distribution. Theoretical analysis shows that the minimax dynamics drive the generated distribution toward the true data distribution, yielding both improved alignment and zero-shot understanding from few-shot prompts. Empirically, SGALM achieves state-of-the-art results on GSM8K, ARC-Challenge, and MBPP, and demonstrates positive scaling when using synthetic data while mitigating over-fitting and mode collapse. Additionally, SGALM functions as a robust synthetic data engine, producing high-quality data that improves downstream fine-tuning beyond real-data baselines. The approach offers a fully grounded, reward-free alternative to RLHF and iterative self-play, with implications for scalable, domain-agnostic alignment and data generation.

Abstract

Fine-tuning large language models (LLMs) for alignment typically relies on supervised fine-tuning or reinforcement learning from human feedback, both limited by the cost and scarcity of high-quality annotations. Recent self-play and synthetic data approaches reduce this dependence but often rely on heuristic assumptions or ungrounded self-evaluation, which can cause bias accumulation and performance drift. In this paper, we propose Self-Generative Adversarial LLM (SGALM), a unified fine-tuning framework that formulates alignment as a generative adversarial game within a single LLM. SGALM jointly evolves generation and discrimination capabilities without external reward models. Theoretical and empirical results demonstrate that SGALM achieves state-of-the-art performance, serves as an effective alignment algorithm and a robust synthetic data engine.

Self-Generative Adversarial Fine-Tuning for Large Language Models

TL;DR

SGALM reframes LLM alignment as a self-contained adversarial game in which a single model alternately acts as generator and discriminator via in-context learning and its own output distribution. Theoretical analysis shows that the minimax dynamics drive the generated distribution toward the true data distribution, yielding both improved alignment and zero-shot understanding from few-shot prompts. Empirically, SGALM achieves state-of-the-art results on GSM8K, ARC-Challenge, and MBPP, and demonstrates positive scaling when using synthetic data while mitigating over-fitting and mode collapse. Additionally, SGALM functions as a robust synthetic data engine, producing high-quality data that improves downstream fine-tuning beyond real-data baselines. The approach offers a fully grounded, reward-free alternative to RLHF and iterative self-play, with implications for scalable, domain-agnostic alignment and data generation.

Abstract

Fine-tuning large language models (LLMs) for alignment typically relies on supervised fine-tuning or reinforcement learning from human feedback, both limited by the cost and scarcity of high-quality annotations. Recent self-play and synthetic data approaches reduce this dependence but often rely on heuristic assumptions or ungrounded self-evaluation, which can cause bias accumulation and performance drift. In this paper, we propose Self-Generative Adversarial LLM (SGALM), a unified fine-tuning framework that formulates alignment as a generative adversarial game within a single LLM. SGALM jointly evolves generation and discrimination capabilities without external reward models. Theoretical and empirical results demonstrate that SGALM achieves state-of-the-art performance, serves as an effective alignment algorithm and a robust synthetic data engine.
Paper Structure (40 sections, 6 theorems, 22 equations, 5 figures, 3 tables, 1 algorithm)

This paper contains 40 sections, 6 theorems, 22 equations, 5 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Our Method
  4. Generation
  5. Discrimination
  6. Updating
  7. Theoretical Analysis
  8. Distribution Alignment via Generative Adversarial Game
  9. From Few-Shot Generation to Zero-Shot Understanding
  10. Comparison with Existing Methods
  11. Empirical Results
  12. Experimental Setup
  13. Main Results
  14. Ablations and Analysis
  15. Diverse Synthetic Samples Mitigates Over-Fitting
  16. ...and 25 more sections

Key Result

Proposition 4.2

For a fixed generator distribution $p_{G}(z)$, the optimal discriminator $D^*(z)$ trained via eq:d1 is making discrimination by: $D^*(z)=p_T(z)/p_T(z) + p_G(z)$.

Figures (5)

  • Figure 1: Illustration of the proposed self-generative adversarial fine-tuning framework.
  • Figure 2: Test accuracy across training iterations on GSM8K, ARC-Challenge, and MBPP for SGALM, its variants, and baseline methods.
  • Figure 3: $\log(p^{\text{real}}_\theta)$ vs Iteration of real and generated samples resp.
  • Figure 4: Distribution of $p^{\text{real}}_\theta$ of seen real samples (training set), unseen real samples (test set), and generated samples.
  • Figure 5: $\text{KL}(p^G_{\theta^i}||p^G_{\theta^{i-1}})$ for each iteration $i$.

Theorems & Definitions (9)

  • Proposition 4.2
  • Proposition 4.3
  • Theorem 4.5
  • Proposition 2.1
  • proof
  • Proposition 2.2
  • proof
  • Theorem 2.3
  • proof