Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Direct Preference Optimization with Rating Information: Practical Algorithms and Provable Gains

Luca Viano, Ruida Zhou, Yifan Sun, Mahdi Namazifar, Volkan Cevher, Shoham Sabach, Mohammad Ghavamzadeh

TL;DR

This work extends direct preference optimization by incorporating rating gap information into alignment with large language models. It introduces three methods—RDPO, RIPO, and ML-RDPO—that combine ranking signals with rating gaps, deriving RDPO/RIPO from a linear reward combination and ML-RDPO from a maximum-likelihood framework. Theoretical results show acceleration when rating gaps are accurate and robustness under noise, with ML-RDPO offering additional benefits when rating data are partially observed or heterogeneous. Empirically, the methods outperform traditional ranking-only DPO-style approaches across multiple models and benchmarks, validating both the theoretical gains and practical applicability for improved, data-efficient alignment. The findings also provide practical guidance on hyperparameter choices and demonstrate resilience to imperfect rating information.

Abstract

The class of direct preference optimization (DPO) algorithms has emerged as a promising approach for solving the alignment problem in foundation models. These algorithms work with very limited feedback in the form of pairwise preferences and fine-tune models to align with these preferences without explicitly learning a reward model. While the form of feedback used by these algorithms makes the data collection process easy and relatively more accurate, its ambiguity in terms of the quality of responses could have negative implications. For example, it is not clear if a decrease (increase) in the likelihood of preferred (dispreferred) responses during the execution of these algorithms could be interpreted as a positive or negative phenomenon. In this paper, we study how to design algorithms that can leverage additional information in the form of rating gap, which informs the learner how much the chosen response is better than the rejected one. We present new algorithms that can achieve faster statistical rates than DPO in presence of accurate rating gap information. Moreover, we theoretically prove and empirically show that the performance of our algorithms is robust to inaccuracy in rating gaps. Finally, we demonstrate the solid performance of our methods in comparison to a number of DPO-style algorithms across a wide range of LLMs and evaluation benchmarks.

Direct Preference Optimization with Rating Information: Practical Algorithms and Provable Gains

TL;DR

This work extends direct preference optimization by incorporating rating gap information into alignment with large language models. It introduces three methods—RDPO, RIPO, and ML-RDPO—that combine ranking signals with rating gaps, deriving RDPO/RIPO from a linear reward combination and ML-RDPO from a maximum-likelihood framework. Theoretical results show acceleration when rating gaps are accurate and robustness under noise, with ML-RDPO offering additional benefits when rating data are partially observed or heterogeneous. Empirically, the methods outperform traditional ranking-only DPO-style approaches across multiple models and benchmarks, validating both the theoretical gains and practical applicability for improved, data-efficient alignment. The findings also provide practical guidance on hyperparameter choices and demonstrate resilience to imperfect rating information.

Abstract

The class of direct preference optimization (DPO) algorithms has emerged as a promising approach for solving the alignment problem in foundation models. These algorithms work with very limited feedback in the form of pairwise preferences and fine-tune models to align with these preferences without explicitly learning a reward model. While the form of feedback used by these algorithms makes the data collection process easy and relatively more accurate, its ambiguity in terms of the quality of responses could have negative implications. For example, it is not clear if a decrease (increase) in the likelihood of preferred (dispreferred) responses during the execution of these algorithms could be interpreted as a positive or negative phenomenon. In this paper, we study how to design algorithms that can leverage additional information in the form of rating gap, which informs the learner how much the chosen response is better than the rejected one. We present new algorithms that can achieve faster statistical rates than DPO in presence of accurate rating gap information. Moreover, we theoretically prove and empirically show that the performance of our algorithms is robust to inaccuracy in rating gaps. Finally, we demonstrate the solid performance of our methods in comparison to a number of DPO-style algorithms across a wide range of LLMs and evaluation benchmarks.
Paper Structure (52 sections, 17 theorems, 155 equations, 11 figures, 1 table, 2 algorithms)

This paper contains 52 sections, 17 theorems, 155 equations, 11 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Notation.
  4. Algorithms for Alignment with Ranking and Rating Information
  5. Ratings Direct Preference Optimization (RDPO)
  6. Maximum-Likelihood-based RDPO (ML-RDPO)
  7. Theoretical results
  8. Theoretical Guarantees for RDPO
  9. Acceleration.
  10. Robustness.
  11. Principled choice of $\beta$ and $\beta_1$ in practice.
  12. Theoretical Guarantees for ML-RDPO
  13. Acceleration.
  14. Robustness.
  15. Comparison with RDPO.
  16. ...and 37 more sections

Key Result

Lemma 3.1

Let us assume that the rating gaps $\Delta^i_{\hat{r}} | x_i, a_i, a'_i$ are normally distributed with variance $\mathbb{V}$, and $z_i$, i.e. the preference bits, and $\Delta^i_{\hat{r}}$ are conditionally independent given $x_i,a_i,a'_i$. Then, it holds that where $\Delta^i_r := r(x_i,a^+_i) - r(x_i,a^-_i)$.

Figures (11)

  • Figure 1: Win rates averaged for $\pi_{\mathrm{ref}} \in \left\{{\texttt{Llama3.1-8B}, \texttt{Zephyr-7B}, \texttt{Mistral-7B}}\right\}$. Our methods are labelled in blue.
  • Figure 2: Win rates against GPT4, judged by Claude-Sonnet-3.5 v2, in AlpacaEval and ArenaHard.
  • Figure 3: Robustness to inaccurate ratings experiments
  • Figure 4: ML-RDPO on ultrafeedback with partial ratings observation, $\pi_{\mathrm{ref}} = \texttt{Mistral-7B}$.
  • Figure 5: Comparison between $D_{\mathrm{KL}}$ and $D_{\chi^2}$ and their differences for Bernoulli random variables.
  • ...and 6 more figures

Theorems & Definitions (29)

  • Lemma 3.1
  • Theorem 4.3
  • Theorem 4.4
  • Lemma B.1
  • proof
  • proof
  • Theorem C.1
  • proof
  • Lemma C.1
  • proof
  • ...and 19 more