Eliminating Biased Length Reliance of Direct Preference Optimization via Down-Sampled KL Divergence

TL;DR

The paper identifies verbosity in Direct Preference Optimization (DPO) as arising from an algorithmic length reliance in the KL-based reward differences between chosen and rejected sequences. It introduces SamPO, a down-sampled KL approach that computes token-level divergences on a probabilistic subset of tokens to remove length bias. Across three foundation models and multiple benchmarks (including UltraFeedback, HH-RLHF, and TL;DR), SamPO reduces verbosity and improves accuracy by 5–12% relative to DPO, with iterative variants offering further gains and human evaluators confirming its effectiveness. The work provides a simple, practical method to debias DPO without deploying a separate reward model, potentially simplifying scalable alignment of large language systems.

Abstract

Direct Preference Optimization (DPO) has emerged as a prominent algorithm for the direct and robust alignment of Large Language Models (LLMs) with human preferences, offering a more straightforward alternative to the complex Reinforcement Learning from Human Feedback (RLHF). Despite its promising efficacy, DPO faces a notable drawback: "verbosity", a common over-optimization phenomenon also observed in RLHF. While previous studies mainly attributed verbosity to biased labels within the data, we propose that the issue also stems from an inherent algorithmic length reliance in DPO. Specifically, we suggest that the discrepancy between sequence-level Kullback-Leibler (KL) divergences between chosen and rejected sequences, used in DPO, results in overestimated or underestimated rewards due to varying token lengths. Empirically, we utilize datasets with different label lengths to demonstrate the presence of biased rewards. We then introduce an effective downsampling approach, named SamPO, to eliminate potential length reliance. Our experimental evaluations, conducted across three LLMs of varying scales and a diverse array of conditional and open-ended benchmarks, highlight the efficacy of SamPO in mitigating verbosity, achieving improvements of 5% to 12% over DPO through debaised rewards. Our codes can be accessed at: https://github.com/LuJunru/SamPO/.

Figures (6)

• Figure 1: Down-Sampling strategy helps mitigate the potential length reliance, and thus improves DPO.
• Figure 2: The inequality of pairwise responses, illustrated by typical examples, forces DPO to overestimate or underestimate actual rewards. In the upper sub-figure (a), we present DPO's chosen reward $\sum\log\frac{\pi_{\theta}(y_w|x)}{\pi_{ref}(y_w|x)}$ and rejected reward $\sum\log\frac{\pi_{\theta}(y_l|x)}{\pi_{ref}(y_l|x)}$ with red and purple curves, respectively. Each response's reward is sequence-level KL divergence, summarizing from token-level log probability ratios (the green and blue bars). Therefore, the difference gap between two lines draws the implicit reward target in DPO, as shown in Eq. \ref{['eqa:dpo_2_t']}. Similarly, we show averagely normalized DPO in left bottom sub-figure (b), and show our SamPO in right bottom sub-figure (c).
• Figure 3: Trends of DPO's implicit reward (Eq. \ref{['eqa:dpo_2_t']}), when fine-tuned with UltraFeedback-long, -short and -all sets. Three debiased rewards are produced by our SamPO.
• Figure 4: We show how the policy model’s response length changes on AlpacEval2 as the test performance improves over 3 epochs of training. The epoch number increases from left to right along the curve.
• Figure 5: Case examples of AlpacaEval2, generated by Llama3-8B-Instruct-SamPO and -DPO. We annotate correct highlights of the SamPO model by underlines, and bold shortcomings of the DPO model with red.