Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the Role of Preference Variance in Preference Optimization

Jiacheng Guo, Zihao Li, Jiahao Qiu, Yue Wu, Mengdi Wang

TL;DR

This work introduces Preference Variance (PVar) as a principled measure of how much a prompt elicits diverse model preferences, and proves that the online DPO gradient is bounded by PVar. It then shows how offline PVar estimates, obtained with a reward model, can guide data selection to favor informative, high-variance prompts. Empirical results across multiple models and alignment benchmarks demonstrate that high-PVar data yields larger learning signals, faster convergence, and better alignment, with substantial data-efficiency gains when training on just the top 10% of prompts. The findings highlight a practical pathway to reduce human annotation costs in LLM alignment without sacrificing performance, by leveraging variance-based data selection robust to reward-model quality.

Abstract

Direct Preference Optimization (DPO) has emerged as an important approach for learning from human preferences in aligning large language models (LLMs). However, collecting human preference data is costly and inefficient, motivating methods to reduce the required annotations. In this work, we investigate the impact of \emph{preference variance} (PVar), which measures the variance in model preferences when comparing pairs of responses, on the effectiveness of DPO training. We provide a theoretical insight by establishing an upper bound on the DPO gradient norm for any given prompt, showing it is controlled by the PVar of that prompt. This implies that prompts with low PVar can only produce small gradient updates, making them less valuable for learning. We validate this finding by fine-tuning LLMs with preferences generated by a reward model, evaluating on two benchmarks (AlpacaEval 2.0 and Arena-Hard). Experimental results demonstrate that prompts with higher PVar outperform randomly selected prompts or those with lower PVar. We also show that our PVar-based selection method is robust, when using smaller reward models (1B, 3B) for selection. Notably, in a separate experiment using the original human annotations from the UltraFeedback dataset, we found that training on only the top 10\% of prompts with the highest PVar yields better evaluation performance than training on the full dataset, highlighting the importance of preference variance in identifying informative examples for efficient LLM alignment.

On the Role of Preference Variance in Preference Optimization

TL;DR

This work introduces Preference Variance (PVar) as a principled measure of how much a prompt elicits diverse model preferences, and proves that the online DPO gradient is bounded by PVar. It then shows how offline PVar estimates, obtained with a reward model, can guide data selection to favor informative, high-variance prompts. Empirical results across multiple models and alignment benchmarks demonstrate that high-PVar data yields larger learning signals, faster convergence, and better alignment, with substantial data-efficiency gains when training on just the top 10% of prompts. The findings highlight a practical pathway to reduce human annotation costs in LLM alignment without sacrificing performance, by leveraging variance-based data selection robust to reward-model quality.

Abstract

Direct Preference Optimization (DPO) has emerged as an important approach for learning from human preferences in aligning large language models (LLMs). However, collecting human preference data is costly and inefficient, motivating methods to reduce the required annotations. In this work, we investigate the impact of \emph{preference variance} (PVar), which measures the variance in model preferences when comparing pairs of responses, on the effectiveness of DPO training. We provide a theoretical insight by establishing an upper bound on the DPO gradient norm for any given prompt, showing it is controlled by the PVar of that prompt. This implies that prompts with low PVar can only produce small gradient updates, making them less valuable for learning. We validate this finding by fine-tuning LLMs with preferences generated by a reward model, evaluating on two benchmarks (AlpacaEval 2.0 and Arena-Hard). Experimental results demonstrate that prompts with higher PVar outperform randomly selected prompts or those with lower PVar. We also show that our PVar-based selection method is robust, when using smaller reward models (1B, 3B) for selection. Notably, in a separate experiment using the original human annotations from the UltraFeedback dataset, we found that training on only the top 10\% of prompts with the highest PVar yields better evaluation performance than training on the full dataset, highlighting the importance of preference variance in identifying informative examples for efficient LLM alignment.
Paper Structure (43 sections, 6 theorems, 25 equations, 3 figures, 4 tables)

This paper contains 43 sections, 6 theorems, 25 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. DPO and Its Variants.
  4. Theoretical Analysis in RLHF.
  5. Preliminaries
  6. Notations.
  7. Preference Probability with Rewards and DPO Objective.
  8. Preference Variance (PVar).
  9. Estimating PVar in Practice.
  10. Theoretical Analysis of DPO Gradient and PVar
  11. Gradient of DPO Loss.
  12. Online Gradient Bound via Online PVar.
  13. Proof Sketch for Theorem \ref{['thm:dpo_online']}.
  14. Bridging Offline Selection and Online Dynamics.
  15. Proof Sketch for Theorem \ref{['thm:dpo_offline']}.
  16. ...and 28 more sections

Key Result

Theorem 4.1

For parameters $\theta \in \mathbb{R}^p$ and a specific input $x \in \mathcal{X}$, the norm of the DPO loss gradient is upper bounded by: where $\text{PVar}_\theta[x]$ is the online preference variance for prompt $x$ computed with the policy $\pi_\theta$, and $C(x, \theta)$ is a term dependent on the model's Jacobian norm and response length, defined above.

Figures (3)

  • Figure 1: Comparison of prompts with different Preference Variance (PVar). Left: A prompt with low PVar (e.g., 'Are you an AI assistant?'). Responses to such prompts are often semantically similar (e.g., minor variations of an affirmative answer), leading to minimal preference differences, low PVar, and a weak training signal. Right: A prompt with high PVar (e.g., 'How to hack a computer?'). This type of prompt can generate a wide range of responses, from harmful compliance to proper refusals, creating strong preference differences, high PVar, and consequently stronger optimization gradients during DPO training.
  • Figure 2: Left and middle: Distribution of Preference Variance (PVar) across prompts from two different datasets: Chatbot Arena Conversation and UltraFeedback. Each PVar is calculated using 5 responses generated by Llama 3.1-8B-Instruct. The distributions show a wide spread of PVar values, indicating significant variation in the informativeness of prompts, with a substantial portion exhibiting moderate-to-high PVar values. Right: Training loss curves for models fine-tuned on different data subsets selected based on PVar. The Top 50% subset (green) demonstrates faster convergence and reaches a lower final loss compared to Random 50% (red) and Bottom 50% (blue) selections, indicating more efficient learning from high-PVar training examples.
  • Figure 3: Training margin curves for models fine-tuned on different data subsets selected based on PVar. The margin represents the difference in model confidence between preferred and rejected responses during training. Models trained on the Top 50% subset (green) show the fastest margin increase and achieve the highest final margin values, while the Bottom 50% subset (blue) exhibits slower margin growth and lower final margins. The Random 50% selection (red) demonstrates intermediate performance, confirming that high-PVar examples facilitate more effective preference learning.

Theorems & Definitions (11)

  • Theorem 4.1: PVar Bounds the DPO Gradient
  • Theorem 4.2: Offline-to-Online Gradient Bound
  • Theorem B.1: Formal version of Theorem \ref{['thm:dpo_online']}
  • proof
  • Lemma C.1: Variance Difference by Measure Change
  • proof
  • Lemma C.2: PVar Difference by Reward Change
  • proof
  • Lemma C.3: TV distance of Product Measures
  • proof
  • ...and 1 more