Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Self-Augmented Preference Optimization: Off-Policy Paradigms for Language Model Alignment

Yueqin Yin, Zhendong Wang, Yujia Xie, Weizhu Chen, Mingyuan Zhou

TL;DR

The paper tackles the limitations of relying on static, pre-collected paired preferences for LLM alignment by introducing Self-Augmented Preference Optimization (SAPO), an online, off-policy training paradigm that uses an Exponential Moving Average (EMA) model and a replay buffer to self-augment training data. SAPO adds segment-level supervision, regenerates targeted rejected segments via the EMA, and updates data through a FIFO buffer, creating a curriculum-like progression that mitigates training volatility. Empirical results on LLaMA-3-8B and Mistral-7B across Open LLM Leaderboard, IFEval, MT-Bench, and AlpacaEval 2.0 show SAPO matching or surpassing offline baselines like DPO and ORPO, and outperforming offline SPIN in most settings. The approach reduces dependency on annotated paired data and external reward models, offering a scalable, data-efficient path toward robust LLM alignment with human preferences, with code available at the provided GitHub link.

Abstract

Traditional language model alignment methods, such as Direct Preference Optimization (DPO), are limited by their dependence on static, pre-collected paired preference data, which hampers their adaptability and practical applicability. To overcome this limitation, we introduce Self-Augmented Preference Optimization (SAPO), an effective and scalable training paradigm that does not require existing paired data. Building on the self-play concept, which autonomously generates negative responses, we further incorporate an off-policy learning pipeline to enhance data exploration and exploitation. Specifically, we employ an Exponential Moving Average (EMA) model in conjunction with a replay buffer to enable dynamic updates of response segments, effectively integrating real-time feedback with insights from historical data. Our comprehensive evaluations of the LLaMA3-8B and Mistral-7B models across benchmarks, including the Open LLM Leaderboard, IFEval, AlpacaEval 2.0, and MT-Bench, demonstrate that SAPO matches or surpasses established offline contrastive baselines, such as DPO and Odds Ratio Preference Optimization, and outperforms offline self-play methods like SPIN. Our code is available at https://github.com/yinyueqin/SAPO

Self-Augmented Preference Optimization: Off-Policy Paradigms for Language Model Alignment

TL;DR

The paper tackles the limitations of relying on static, pre-collected paired preferences for LLM alignment by introducing Self-Augmented Preference Optimization (SAPO), an online, off-policy training paradigm that uses an Exponential Moving Average (EMA) model and a replay buffer to self-augment training data. SAPO adds segment-level supervision, regenerates targeted rejected segments via the EMA, and updates data through a FIFO buffer, creating a curriculum-like progression that mitigates training volatility. Empirical results on LLaMA-3-8B and Mistral-7B across Open LLM Leaderboard, IFEval, MT-Bench, and AlpacaEval 2.0 show SAPO matching or surpassing offline baselines like DPO and ORPO, and outperforming offline SPIN in most settings. The approach reduces dependency on annotated paired data and external reward models, offering a scalable, data-efficient path toward robust LLM alignment with human preferences, with code available at the provided GitHub link.

Abstract

Traditional language model alignment methods, such as Direct Preference Optimization (DPO), are limited by their dependence on static, pre-collected paired preference data, which hampers their adaptability and practical applicability. To overcome this limitation, we introduce Self-Augmented Preference Optimization (SAPO), an effective and scalable training paradigm that does not require existing paired data. Building on the self-play concept, which autonomously generates negative responses, we further incorporate an off-policy learning pipeline to enhance data exploration and exploitation. Specifically, we employ an Exponential Moving Average (EMA) model in conjunction with a replay buffer to enable dynamic updates of response segments, effectively integrating real-time feedback with insights from historical data. Our comprehensive evaluations of the LLaMA3-8B and Mistral-7B models across benchmarks, including the Open LLM Leaderboard, IFEval, AlpacaEval 2.0, and MT-Bench, demonstrate that SAPO matches or surpasses established offline contrastive baselines, such as DPO and Odds Ratio Preference Optimization, and outperforms offline self-play methods like SPIN. Our code is available at https://github.com/yinyueqin/SAPO
Paper Structure (32 sections, 6 equations, 2 figures, 5 tables, 1 algorithm)

This paper contains 32 sections, 6 equations, 2 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Definition of Learning Paradigms in LLM Alignment
  4. Offline Off-Policy Contrastive Preference Learning Algorithms
  5. Self-Play Fine-Tuning
  6. Self-Augmented Preference Optimization
  7. Segment-Level Supervision in SAPO.
  8. Off-Policy Learning Setting.
  9. Experiments
  10. Experimental Setup.
  11. Baselines.
  12. Datasets and Base Models.
  13. Evaluation Benchmarks.
  14. Training Details.
  15. Benchmark Performance
  16. ...and 17 more sections

Figures (2)

  • Figure 1: Given a prompt $x_1$ and chosen response $y_1^+$. This response is segmented into $A$, $B$, and $C$. Using the prompt with segment $A$, the EMA model generates a new segment $B'$. Together, segments $A$, $B'$, and $C$ form the rejected response $y_1^-$, which is appended to the replay buffer. Random tuples are sampled from this buffer to train the policy network, subsequently updating the EMA weights.
  • Figure 2: Ablation of training epochs on LLaMA-3-8B using ORPO across multiple benchmarks.