TSO: Self-Training with Scaled Preference Optimization

TL;DR

TSO addresses the challenge of aligning LLMs to human preferences by eliminating the need for a separate reward model and instead using self-training guided by a model matrix to amplify response diversity. It combines three components—model matrix-based data construction, human/AI evaluation correction, and a Mini-Batches Iterative DPO training regime with a dual clip reward loss—to balance diversity, validity, and adaptability in preference data. The approach yields consistent gains over strong baselines across multiple alignment benchmarks, demonstrating the practical viability of diversified, corrected preference data and iterative training in the alignment domain. These findings offer actionable insights for constructing diverse, valid, and adaptable preference datasets and for training alignment-focused models without heavy reward-model infrastructure.

Abstract

Enhancing the conformity of large language models (LLMs) to human preferences remains an ongoing research challenge. Recently, offline approaches such as Direct Preference Optimization (DPO) have gained prominence as attractive options due to offering effective improvement in simple, efficient, and stable without interactions with reward models. However, these offline preference optimization methods highly rely on the quality of pairwise preference samples. Meanwhile, numerous iterative methods require additional training of reward models to select positive and negative samples from the model's own generated responses for preference learning. Furthermore, as LLMs' capabilities advance, it is quite challenging to continuously construct high-quality positive and negative preference instances from the model's outputs due to the lack of diversity. To tackle these challenges, we propose TSO, or Self-Training with Scaled Preference Optimization, a framework for preference optimization that conducts self-training preference learning without training an additional reward model. TSO enhances the diversity of responses by constructing a model matrix and incorporating human preference responses. Furthermore, TSO introduces corrections for model preference errors through human and AI feedback. Finally, TSO adopts iterative and dual clip reward strategies to update the reference model and its responses, adaptively adjusting preference data and balancing the optimization process. Experimental results demonstrate that TSO outperforms existing mainstream methods on various alignment evaluation benchmarks, providing practical insight into preference data construction and model training strategies in the alignment domain.

Figures (6)

• Figure 1: TSO first samples responses from the model matrix, ensuring the diversity of the positive and negative response datasets. Then, it uses feedback from humans or AI to correct validity bias. Finally, it employs the Mini-Batch Iterative DPO and Dual Clip Reward Loss strategies for DPO training. The above steps are repeated $N$ times.
• Figure 2: Model Matrix Instructions Construction. For cross-version augment, the model utilizes inferences from the older version of the model as candidate negative responses. For cross-scale augment, the model utilizes inferences from a smaller model as candidate negative responses. Meanwhile, the latest and largest model's inferences are used as candidate positive responses.
• Figure 3: DPO stands for using $\mathcal{L}_{DPO}$ . The blue line signifies the rewards obtained from positive responses, i.e. , $\beta\log \frac{\pi_{\theta}(y_w|x)}{\pi_{ref}(y_w|x)}$ , while the red line indicates the rewards obtained from negative responses, i.e. , $\beta\log \frac{\pi_{\theta}(y_l|x)}{\pi_{ref}(y_l|x)}$.
• Figure 4: CLIP represents using $\mathcal{L}_{dual-clip}$.
• Figure 5: Data distribution. For each question, we generate 64 different answers and scores. Every point stands for a question."delta" represents the skewness of scores distribution, "kurtosis_val" represents the normalized kurtosis of scores distribution.

Theorems & Definitions (1)

• Definition 1