Towards Improved Preference Optimization Pipeline: from Data Generation to Budget-Controlled Regularization

TL;DR

This work aims to improve the preference optimization pipeline by taking a closer look at preference data generation and training regularization techniques, and proposes an iterative pairwise ranking mechanism that derives preference ranking of completions using pairwise comparison signals.

Abstract

Direct Preference Optimization (DPO) and its variants have become the de facto standards for aligning large language models (LLMs) with human preferences or specific goals. However, DPO requires high-quality preference data and suffers from unstable preference optimization. In this work, we aim to improve the preference optimization pipeline by taking a closer look at preference data generation and training regularization techniques. For preference data generation, we demonstrate that existing scoring-based reward models produce unsatisfactory preference data and perform poorly on out-of-distribution tasks. This significantly impacts the LLM alignment performance when using these data for preference tuning. To ensure high-quality preference data generation, we propose an iterative pairwise ranking mechanism that derives preference ranking of completions using pairwise comparison signals. For training regularization, we observe that preference optimization tends to achieve better convergence when the LLM predicted likelihood of preferred samples gets slightly reduced. However, the widely used supervised next-word prediction regularization strictly prevents any likelihood reduction of preferred samples. This observation motivates our design of a budget-controlled regularization formulation. Empirically we show that combining the two designs leads to aligned models that surpass existing SOTA across two popular benchmarks.

Figures (7)

• Figure 1: Overview for DPO pipeline. Preference data generation: existing scoring-based methods select preferred and dispreferred completions based on a single score, our proposed iterative pairwise ranking uses pairwise comparison signals to construct preference data. Regularizing preference optimization: we propose a budget-controlled regularization that balances training stability and model alignment performance.
• Figure 2: Training progresses of DPO and DPOP. (a) Reward margin: Measures the difference in rewards between preferred and dispreferred completions, which is the main objective in DPO training. (b) Reward accuracy: Shows the percentage of preferred completions that have higher rewards than their dispreferred ones. (c) Log probability: Indicates the average log-likelihood of preferred completions.
• Figure 3: Optimization budget (log-likelihood of preferred completions) versus Alpaca-Eval win rate score. Each point corresponds to a model trained on a particular set of hyperparameters.
• Figure 4: Optimization budget (log-likelihood of preferred completions) versus Alpaca-Eval. (a) DPO versus DPO-BCR: sum of log-likelihood of preferred completions is used. (b) SimPO versus SimPO-BCR: average of log-likelihood of preferred completions is used.
• Figure 5: Statistics of IPR. For IPR(Llama70B) with Llama-3.1-Instruct as base model: (a1), (b1), (c1) and (d1) present the statistics of preference comparisons at all $4$ iterations. For IPR(Llama70B) with Mistral-Instruct as base model: (a2), (b2), (c2) and (d2) present the statistics of preference comparisons at all $4$ iterations.