Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Direct Preference Optimization with an Offset

Afra Amini, Tim Vieira, Ryan Cotterell

TL;DR

This work extends direct preference optimization (DPO) by introducing ODPO, which incorporates an offset that scales with how strongly a preferred output is favored over a dispreferred one. Grounded in a Gumbel-based connection to Bradley–Terry and softmax-margin concepts, ODPO enforces a margin that grows with the reward difference, enabling more data-efficient alignment without reward models or RL. Empirically, ODPO consistently yields better reward–KL tradeoffs and higher win rates than DPO across sentiment control, toxicity reduction, and summarization, especially when preference data are limited. The paper also conducts ablations on the scaling function and offset strength, discusses limitations related to the availability of degree-of-preference data, and addresses ethical considerations in RLHF-style fine-tuning.

Abstract

Direct preference optimization (DPO) is a successful fine-tuning strategy for aligning large language models with human preferences without the need to train a reward model or employ reinforcement learning. DPO, as originally formulated, relies on binary preference data and fine-tunes a language model to increase the likelihood of a preferred response over a dispreferred response. However, not all preference pairs are equal. Sometimes, the preferred response is only slightly better than the dispreferred one. In other cases, the preference is much stronger. For instance, if a response contains harmful or toxic content, the annotator will have a strong preference for that response. In this paper, we propose a generalization of DPO, termed DPO with an offset (ODPO), that does not treat every preference pair equally during fine-tuning. Intuitively, ODPO requires the difference between the likelihood of the preferred and dispreferred response to be greater than an offset value. The offset is determined based on the extent to which one response is preferred over another. Our experiments on various tasks suggest that ODPO significantly outperforms DPO in aligning language models, especially when the number of preference pairs is limited.

Direct Preference Optimization with an Offset

TL;DR

This work extends direct preference optimization (DPO) by introducing ODPO, which incorporates an offset that scales with how strongly a preferred output is favored over a dispreferred one. Grounded in a Gumbel-based connection to Bradley–Terry and softmax-margin concepts, ODPO enforces a margin that grows with the reward difference, enabling more data-efficient alignment without reward models or RL. Empirically, ODPO consistently yields better reward–KL tradeoffs and higher win rates than DPO across sentiment control, toxicity reduction, and summarization, especially when preference data are limited. The paper also conducts ablations on the scaling function and offset strength, discusses limitations related to the availability of degree-of-preference data, and addresses ethical considerations in RLHF-style fine-tuning.

Abstract

Direct preference optimization (DPO) is a successful fine-tuning strategy for aligning large language models with human preferences without the need to train a reward model or employ reinforcement learning. DPO, as originally formulated, relies on binary preference data and fine-tunes a language model to increase the likelihood of a preferred response over a dispreferred response. However, not all preference pairs are equal. Sometimes, the preferred response is only slightly better than the dispreferred one. In other cases, the preference is much stronger. For instance, if a response contains harmful or toxic content, the annotator will have a strong preference for that response. In this paper, we propose a generalization of DPO, termed DPO with an offset (ODPO), that does not treat every preference pair equally during fine-tuning. Intuitively, ODPO requires the difference between the likelihood of the preferred and dispreferred response to be greater than an offset value. The offset is determined based on the extent to which one response is preferred over another. Our experiments on various tasks suggest that ODPO significantly outperforms DPO in aligning language models, especially when the number of preference pairs is limited.
Paper Structure (34 sections, 3 theorems, 21 equations, 6 figures, 6 tables)

This paper contains 34 sections, 3 theorems, 21 equations, 6 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Reward Modeling
  4. Reward Modeling
  5. Pointwise Rewards.
  6. Pairwise Preferences.
  7. Direct Preference Optimization
  8. Designing $\deltar$.
  9. Connection to Softmax Margin.
  10. Experiments
  11. Sentiment Control
  12. Reward Model.
  13. Bootstraping the Preference Data.
  14. Toxicity Control
  15. Summarization
  16. ...and 19 more sections

Key Result

theorem 1

Let $\by_w$ and $\by_l$ be two responses to a prompt $\bx$, and let $\rhattheta(\bx, \by_w)$ and $\rhattheta(\bx, \by_l)$ be their associated estimated rewards. Finally, let $\rtilde_w \sim \gumbel(\rhattheta(\bx, \by_w), 1)$ and $\rtilde_l \sim \gumbel(\rhattheta(\bx, \by_l), 1)$ be Gumbel random where $\pbt(\by_w \succ \by_l \mid \bx)$ is a Bradley--Terry model eq:bt parameterized by $\rhatthe

Figures (6)

  • Figure 1: ODPO takes into account the extent to which one output should be preferred over another. The model has to put more probability mass on the preferred output compared to the dispreferred output by an offset that is determined based on how much the winning output is preferred over the losing output.
  • Figure 2: Steering generated movie reviews towards positive sentiment. Points on the Pareto front are highlighted with a black border. We observe that in all $3$ settings, most (if not all) points on the Pareto front belong to ODPO.
  • Figure 3: Steering generations away from toxic content. We highlight points on the Pareto front with a black border. We observe that, especially when the size of the dataset is small, ODPO manages to reduce the toxicity better than DPO while not diverging too far from the SFT model.
  • Figure 4: Investigating the effect of the offset formulation on the performance of ODPO. Scaling the offset with a $\log$ function helps achieve the highest reward values without diverging too much from the SFT model.
  • Figure 5: Win rates of generations from models fine-tuned with DPO and ODPO against human-written summaries. On average, ODPO achieves a higher win rate---significantly so in temperatures $0$ and $0.5$.
  • ...and 1 more figures

Theorems & Definitions (6)

  • theorem 1
  • proof
  • theorem 2
  • proof
  • theorem 2
  • proof