Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On Extending Direct Preference Optimization to Accommodate Ties

Jinghong Chen, Guangyu Yang, Weizhe Lin, Jingbiao Mei, Bill Byrne

TL;DR

This work extends Direct Preference Optimization (DPO) to handle ties in pairwise judgments by replacing the Bradley-Terry model with Rao-Kupper (RK) and Davidson (D) extensions that explicitly assign probability to ties. The authors derive tie-inclusive objective functions and gradient updates, showing that including tied data can regularize the learned policy with respect to a reference model while avoiding the performance degradation seen when tying data is ignored in standard DPO. Across neural machine translation and summarization, the tie-aware variants DPO-RK and DPO-D demonstrate improved regularization (lower KL to the reference) and can outperform standard DPO when leveraging tied data, especially in translation and mathematical reasoning tasks. The results advocate for incorporating ties rather than discarding them, and the work provides theoretical and empirical insights into why ties regularize policy learning and how to tune the tie-extensions in practice.

Abstract

We derive and investigate two DPO variants that explicitly model the possibility of declaring a tie in pair-wise comparisons. We replace the Bradley-Terry model in DPO with two well-known modeling extensions, by Rao and Kupper and by Davidson, that assign probability to ties as alternatives to clear preferences. Our experiments in neural machine translation and summarization show that explicitly labeled ties can be added to the datasets for these DPO variants without the degradation in task performance that is observed when the same tied pairs are presented to DPO. We find empirically that the inclusion of ties leads to stronger regularization with respect to the reference policy as measured by KL divergence, and we see this even for DPO in its original form. We provide a theoretical explanation for this regularization effect using ideal DPO policy theory. We further show performance improvements over DPO in translation and mathematical reasoning using our DPO variants. We find it can be beneficial to include ties in preference optimization rather than simply discard them, as is done in common practice.

On Extending Direct Preference Optimization to Accommodate Ties

TL;DR

This work extends Direct Preference Optimization (DPO) to handle ties in pairwise judgments by replacing the Bradley-Terry model with Rao-Kupper (RK) and Davidson (D) extensions that explicitly assign probability to ties. The authors derive tie-inclusive objective functions and gradient updates, showing that including tied data can regularize the learned policy with respect to a reference model while avoiding the performance degradation seen when tying data is ignored in standard DPO. Across neural machine translation and summarization, the tie-aware variants DPO-RK and DPO-D demonstrate improved regularization (lower KL to the reference) and can outperform standard DPO when leveraging tied data, especially in translation and mathematical reasoning tasks. The results advocate for incorporating ties rather than discarding them, and the work provides theoretical and empirical insights into why ties regularize policy learning and how to tune the tie-extensions in practice.

Abstract

We derive and investigate two DPO variants that explicitly model the possibility of declaring a tie in pair-wise comparisons. We replace the Bradley-Terry model in DPO with two well-known modeling extensions, by Rao and Kupper and by Davidson, that assign probability to ties as alternatives to clear preferences. Our experiments in neural machine translation and summarization show that explicitly labeled ties can be added to the datasets for these DPO variants without the degradation in task performance that is observed when the same tied pairs are presented to DPO. We find empirically that the inclusion of ties leads to stronger regularization with respect to the reference policy as measured by KL divergence, and we see this even for DPO in its original form. We provide a theoretical explanation for this regularization effect using ideal DPO policy theory. We further show performance improvements over DPO in translation and mathematical reasoning using our DPO variants. We find it can be beneficial to include ties in preference optimization rather than simply discard them, as is done in common practice.
Paper Structure (72 sections, 1 theorem, 35 equations, 7 figures, 17 tables)

This paper contains 72 sections, 1 theorem, 35 equations, 7 figures, 17 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. DPO and the Bradley-Terry Preference Distribution
  4. Bradley-Terry Extensions that Accommodate Ties
  5. Bradley-Terry Extensions that Accommodate Ties
  6. Balancing Wins and Ties.
  7. Incorporating Rao-Kupper and Davidson Models into DPO
  8. DPO-RK and DPO-D Updates
  9. $\nabla \log p_\theta(y_w \succ_x y_l)$:
  10. $\nabla \log p_\theta(y_w \sim_x y_l)$:
  11. Intuition for hyper-parameter $\alpha_{RK}$ and $\nu_{D}$ :
  12. Experiments in Adding Ties to DPO, DPO-RK, DPO-D
  13. Extending Preference Datasets to Include Ties
  14. Adding Ties to DPO - The Regularizing Effects of Ties
  15. Explaining the Regularization Effect of Ties via Ideal DPO Policy Theory
  16. ...and 57 more sections

Key Result

Theorem 1

Assume we are given an aggregated comparison datapoint $(x, y_1, y_2)$ and human ground-truth preference probabilities $\mathbb{P}(y_1 \succ_x y_2)$, $\mathbb{P}(y_1 \succ_x y_2)$, and $\mathbb{P}(y_1 \sim_x y_2)$ which obey the Davidson model with hyper-parameter $\nu_{D}$. Let the reference model or equivalently

Figures (7)

  • Figure 1: Task Performance vs. KL to the reference policy for DPO systems trained on Clear Preference Pairs (DPO(CP), blue) and on Clear Preference Pairs and Tied Pairs (DPO(CP+TP), green). Task Performance is reported in BLEURT for translation tasks on WMT21 ZH-EN and IWSLT17 FR-EN. Summarization performance is reported on TL;DR in terms of PairRM win-rate against human-written summaries. KL is estimated over 256 test set policy samples; $\beta$ is noted for best performing systems. Full details are in Appendix \ref{['app:experimental details and full results']}.
  • Figure 2: KL-Performance frontiers with DPO(CP) in blue, DPO(CP+TP) in green, DPO-RK(CP+TP) in purple, and DPO-D(CP+TP) in orange. Full details in Appendix \ref{['app:Tabulated KL-Performance Results on NMT and Summarization']}. For TL;DR, we additionally report win-rate as judged by GPT-4 in Table \ref{['tab:gpt4-evaluation-summarization-app']}.
  • Figure 3: The clear preference probabilities $P(y_w\succ y_l|x)$ (left) and tie probabilities $P(y_w\sim y_l | x)$ (right) as a function of reward margins $d_\theta(x, y_w, y_l)$ for Bradley-Terry (as used in DPO) (blue), Rao-Kupper (purple) (as used in DPO-RK), and Davidson (orange) (as used in DPO-D). $\alpha_{RK}=\log3$ and $\nu_D=1$ are used in producing these plots.
  • Figure 4: The gradient scale factors for DPO (blue) and DPO-RK (purple) and DPO-D (orange) as a function of reward margins $d_\theta(x, y_w, y_l)$ on clear preference pairs (up) and tied pairs (down).$\alpha_{RK}=\log3$ and $\nu_D=1$ are used in producing these plots.
  • Figure 5: Empirical distribution of preference probabilities under the Bradley-Terry model using the implicit reward function from the trained DPO policy on heldout CPs and TPs. DPO(CP) in blue, and DPO(CP+TP) in orange.
  • ...and 2 more figures

Theorems & Definitions (2)

  • Theorem 1: Simulating Perfect DPO-D Policy
  • proof