Self-Boosting Large Language Models with Synthetic Preference Data

TL;DR

SynPO introduces a self-boosting paradigm for LLM alignment that relies on synthetic preference data generated through a self-prompt generator and a response improver. By iteratively producing diverse prompts, refining model outputs, and filtering based on internal preference gaps, SynPO trains on synthetic data without large-scale human labeling. Empirical results show substantial gains in instruction-following and downstream task performance across Mistral-Base and Llama3-Base, including notable improvements on AlpacaEval 2.0, Arena-Hard, and the Open LLM leaderboard. The approach reduces annotation costs, mitigates alignment tax across iterations, and demonstrates strong potential for scalable, autonomous LLM improvement.

Abstract

Through alignment with human preferences, Large Language Models (LLMs) have advanced significantly in generating honest, harmless, and helpful responses. However, collecting high-quality preference data is a resource-intensive and creativity-demanding process, especially for the continual improvement of LLMs. We introduce SynPO, a self-boosting paradigm that leverages synthetic preference data for model alignment. SynPO employs an iterative mechanism wherein a self-prompt generator creates diverse prompts, and a response improver refines model responses progressively. This approach trains LLMs to autonomously learn the generative rewards for their own outputs and eliminates the need for large-scale annotation of prompts and human preferences. After four SynPO iterations, Llama3-8B and Mistral-7B show significant enhancements in instruction-following abilities, achieving over 22.1% win rate improvements on AlpacaEval 2.0 and ArenaHard. Simultaneously, SynPO improves the general performance of LLMs on various tasks, validated by a 3.2 to 5.0 average score increase on the well-recognized Open LLM leaderboard.

Figures (10)

• Figure 1: Length-controlled win rate on AlpacaEval 2.0 improves with SynPO iterations, approaching GPT-4 level for the base versions of Llama3-8B and Mistral-7B.
• Figure 2: Results on AlpacaEval 2.0 leaderboard. LC and WR represent length-controlled and raw win rate, respectively.
• Figure 3: Overview of SynPO in the $\mathbf{t^{th}}$ iteration. Starting with the previous iteration model $M_{t-1}$, SynPO first learns a response improver $R_t$ to identify discrepancies between model responses ($y^*_{t-1}$) and gold standard responses ($y^*$) on seed data, and learns to refine model responses. Subsequently, on the self-generated prompts $x$ (elaborated in Section \ref{['method:prompt-gen']}), SynPO employs $R_t$ to refine the $M_{t-1}$ responses ($y_{t-1}$) into improved responses ($y_{t-1}^w$). The valid synthetic prompts $x$, refined responses ($y_{t-1}^w$), and initial model $M_{0}$ responses ($y_0$) to form synthetic preference data. These data are incorporated into the synthetic preference dataset for preference optimization, resulting in an updated $M_t$ for the next iteration. The iterative process continually enhances LLM capabilities in instruction-following and task performance.
• Figure 4: Prompt used in SynPO for LLMs to act as self-prompt generators.
• Figure 5: The top 25 most common topics (outer circle) and the top 12 most common intentions (inner circle) in SynPO generated prompts. We aggregate the other topics and intentions to the 'Others' group.