SPACE: Noise Contrastive Estimation Stabilizes Self-Play Fine-Tuning for Large Language Models

TL;DR

SPACE addresses instability in self-play fine-tuning of LLMs caused by gap-based objectives. It introduces Noise Contrastive Estimation to discriminate real vs synthetic responses, decoupling their absolute rewards and ensuring convergence to the real data distribution. Theoretical results prove achievability and stable convergence, and experiments on Mistral-7B and Zephyr-7B show SPACE outperforms SPIN and SFT with less annotations, along with stable, continual improvements. This approach offers a practical, data-efficient path for adapting LLMs to downstream tasks under limited labeled data.

Abstract

Self-play fine-tuning has demonstrated promising abilities in adapting large language models (LLMs) to downstream tasks with limited real-world data. The basic principle is to iteratively refine the model with real samples and synthetic ones generated from itself. However, the existing methods primarily focus on the relative gaps between the rewards for two types of data, neglecting their absolute values. Through theoretical analysis, we identify that the gap-based methods suffer from unstable evolution, due to the potentially degenerated objectives. To address this limitation, we introduce a novel self-play fine-tuning method, namely Self-PlAy via Noise Contrastive Estimation (SPACE), which leverages noise contrastive estimation to capture the real-world data distribution. Specifically, SPACE treats synthetic samples as auxiliary components, and discriminates them from the real ones in a binary classification manner. As a result, SPACE independently optimizes the absolute reward values for each type of data, ensuring a consistently meaningful objective and thereby avoiding the instability issue. Theoretically, we show that the optimal solution of the objective in SPACE aligns with the underlying distribution of real-world data, and SPACE guarantees a provably stable convergence to the optimal distribution. Empirically, we show that SPACE significantly improves the performance of LLMs over various tasks, and outperforms supervised fine-tuning that employs much more real-world samples. Compared to gap-based self-play fine-tuning methods, SPACE exhibits remarkable superiority and stable evolution.

Figures (8)

• Figure 1: The left part (a) depicts the framework of self-play fine-tuning. The right part (b) shows the difference in optimization forces between SPIN and $\textsc{Space}$, where SPIN maximizes the relative gaps between annotated and synthetic data, while $\textsc{Space}$ optimizes two types of data independently.
• Figure 2: The average scores of SPIN and $\textsc{Space}$ at different iterations on tasks from the HuggingFace Open LLM leaderboard.
• Figure 3: The performance comparisons among four self-play fine-tuning methods on Mistral-7B. (a) the average scores over different tasks; (b) the performances on GSM8K; (c) the average ranks over different iterations, where the best rank among iterations $0$ to $4$ is highlighted with a "gold medal".
• Figure 4: Performance comparisons between the model trained with multiple epochs at iteration $0$ (green) and that trained with two epochs at iteration $1$ (red).
• Figure 5: The average score with different sizes of annotated data. The start point denotes the performance of base model.

Theorems & Definitions (4)

• Proposition 1
• Theorem 1
• Theorem 2
• Theorem 3