Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Triplets Better Than Pairs: Towards Stable and Effective Self-Play Fine-Tuning for LLMs

Yibo Wang, Hai-Long Sun, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, Lijun Zhang

TL;DR

A novel Triplet-based Self-Play fIne-tuNing (T-SPIN) method that integrates two key designs, including the entropy constraint into the self-play framework, which is theoretically justified to support reference-free fine-tuning, eliminating the training-generation discrepancy.

Abstract

Recently, self-play fine-tuning (SPIN) has been proposed to adapt large language models to downstream applications with scarce expert-annotated data, by iteratively generating synthetic responses from the model itself. However, SPIN is designed to optimize the current reward advantages of annotated responses over synthetic responses at hand, which may gradually vanish during iterations, leading to unstable optimization. Moreover, the utilization of reference policy induces a misalignment issue between the reward formulation for training and the metric for generation. To address these limitations, we propose a novel Triplet-based Self-Play fIne-tuNing (T-SPIN) method that integrates two key designs. First, beyond current advantages, T-SPIN additionally incorporates historical advantages between iteratively generated responses and proto-synthetic responses produced by the initial policy. Even if the current advantages diminish, historical advantages remain effective, stabilizing the overall optimization. Second, T-SPIN introduces the entropy constraint into the self-play framework, which is theoretically justified to support reference-free fine-tuning, eliminating the training-generation discrepancy. Empirical results on various tasks demonstrate not only the superior performance of T-SPIN over SPIN, but also its stable evolution during iterations. Remarkably, compared to supervised fine-tuning, T-SPIN achieves comparable or even better performance with only 25% samples, highlighting its effectiveness when faced with scarce annotated data.

Triplets Better Than Pairs: Towards Stable and Effective Self-Play Fine-Tuning for LLMs

TL;DR

A novel Triplet-based Self-Play fIne-tuNing (T-SPIN) method that integrates two key designs, including the entropy constraint into the self-play framework, which is theoretically justified to support reference-free fine-tuning, eliminating the training-generation discrepancy.

Abstract

Recently, self-play fine-tuning (SPIN) has been proposed to adapt large language models to downstream applications with scarce expert-annotated data, by iteratively generating synthetic responses from the model itself. However, SPIN is designed to optimize the current reward advantages of annotated responses over synthetic responses at hand, which may gradually vanish during iterations, leading to unstable optimization. Moreover, the utilization of reference policy induces a misalignment issue between the reward formulation for training and the metric for generation. To address these limitations, we propose a novel Triplet-based Self-Play fIne-tuNing (T-SPIN) method that integrates two key designs. First, beyond current advantages, T-SPIN additionally incorporates historical advantages between iteratively generated responses and proto-synthetic responses produced by the initial policy. Even if the current advantages diminish, historical advantages remain effective, stabilizing the overall optimization. Second, T-SPIN introduces the entropy constraint into the self-play framework, which is theoretically justified to support reference-free fine-tuning, eliminating the training-generation discrepancy. Empirical results on various tasks demonstrate not only the superior performance of T-SPIN over SPIN, but also its stable evolution during iterations. Remarkably, compared to supervised fine-tuning, T-SPIN achieves comparable or even better performance with only 25% samples, highlighting its effectiveness when faced with scarce annotated data.
Paper Structure (21 sections, 2 theorems, 20 equations, 7 figures, 4 tables, 1 algorithm)

This paper contains 21 sections, 2 theorems, 20 equations, 7 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Supervised fine-tuning
  4. Self-play fine-tuning
  5. Triplet-based self-play fine-tuning
  6. Experiments
  7. Experimental setups
  8. Experimental results
  9. Ablation studies
  10. Related work
  11. Conclusion and discussion
  12. Mathematical derivations
  13. Proof of Proposition \ref{['prop:triplet:opponent_player']}
  14. Proof of Theorem \ref{['theorem:gradient']}
  15. More details on experiments
  16. ...and 6 more sections

Key Result

Proposition 1

Given a general confidence function $c_{t+1}$, the optimal policy of the opponent player in eq:triplet:opponent_player takes the form of where $\mathcal{Y}$ denotes the set that contains all possible responses for the prompt $\mathbf{x}$.

Figures (7)

  • Figure 1: Comparisons of three strategies: (a) supervised fine-tuning requires large amounts of annotated data to train $\pi_\theta$; (b) self-play fine-tuning operates with limited annotated data and iteratively generated samples, and employs the previous policy $\pi_{\theta_t}$ as a reference for updates; (c) triplet-based self-play fine-tuning employs triplet inputs, i.e., annotated data, synthetic samples, and proto-synthetic ones from the initial policy $\pi_{\theta_0}$, and updates $\pi_\theta$ without auxiliaries from any reference policies.
  • Figure 2: Performance (%) comparisons between $\mathtt{T}\hbox{-}\mathtt{SPIN}$ and $\mathtt{SPIN}$ on two tasks: GSM8K and IFEval over $5$ iterations. The average scores over $10$ different tasks are also illustrated in the right panel.
  • Figure 3: Comparisons between $\mathtt{SPIN}$ and $\mathtt{T}\hbox{-}\mathtt{SPIN}$ at iteration $1$: (a) Training dynamics (including rewards and log-likelihoods of $\mathbf{y}$ and $\mathbf{y}'$) and generation statistic (i.e., associations between rewards and log-likelihoods) measured on the training set for iteration $1$ of $\mathtt{SPIN}$; (b) Training dynamics and generation statistic of $\mathtt{T}\hbox{-}\mathtt{SPIN}$. For brevity, we denote $r_\mathbf{y} = r(\mathbf{y}|\mathbf{x})$ and $\log \pi_{\mathbf{y}} = \log \pi (\mathbf{y}|\mathbf{x})$.
  • Figure 4: Performance comparisons over different settings: (a) comparison between $\mathtt{T}\hbox{-}\mathtt{SPIN}$ and SFT with varying amounts of annotated data; (b) performances of $\mathtt{T}\hbox{-}\mathtt{SPIN}$ versus the variant without historical advantage ($\mathtt{w}\hbox{/}\mathtt{o~H}\hbox{-}\mathtt{A}$); (c) robustness analysis with respect to hyperparameters $\alpha$ and $\beta$.
  • Figure 5: Performance (%) comparisons between $\mathtt{T}\hbox{-}\mathtt{SPIN}$ and $\mathtt{w}\hbox{/}\mathtt{o~H}\hbox{-}\mathtt{A}$ on GSM8K, MATH and IFEval.
  • ...and 2 more figures

Theorems & Definitions (2)

  • Proposition 1
  • Theorem 1