Preference Optimization by Estimating the Ratio of the Data Distribution

TL;DR

We recast direct preference optimization (DPO) as likelihood-ratio ratio matching and introduce Bregman Preference Optimization (BPO), a general framework over ratio-based divergences with a family of tractable objectives. BPO defines data and model ratios, minimizes a Bregman divergence $D_h(R_{data}||R_{\theta})$, and recovers DPO as a special case when $h$ corresponds to logistic regression; additional gradient-scaling via Scaled Basu’s Power Divergence (SBA) stabilizes training. Theoretical results guarantee that, with sufficient capacity, minimizing $\mathcal{L}^{h}_{BPO}$ recovers the target policy, while SBA provides robust gradient scales across $\lambda$. Empirically, BPO improves both fidelity (win rate) and diversity (entropy) over DPO and other probabilistic DPO variants in dialogue and TL;DR summarization, achieving state-of-the-art performance on AlpacaEval2 with Llama-3-8B-Instruct and demonstrating strong generalization to larger backbones and external benchmarks.

Abstract

Direct preference optimization (DPO) is widely used as a simple and stable method for aligning large language models (LLMs) with human preferences. This paper investigates a generalized DPO loss that enables a policy model to match the target policy from a likelihood ratio estimation perspective. The ratio of the target policy provides a unique identification of the policy distribution without relying on reward models or partition functions. This allows the generalized loss to retain both simplicity and theoretical guarantees, which prior work such as $f$-PO fails to achieve simultaneously. We propose Bregman preference optimization (BPO), a generalized framework for ratio matching that provides a family of objective functions achieving target policy optimality. BPO subsumes DPO as a special case and offers tractable forms for all instances, allowing implementation with a few lines of code. We further develop scaled Basu's power divergence (SBA), a gradient scaling method that can be used for BPO instances. The BPO framework complements other DPO variants and is applicable to target policies defined by these variants. In experiments, unlike other probabilistic loss extensions such as $f$-DPO or $f$-PO, which exhibit a trade-off between generation fidelity and diversity, instances of BPO improve both win rate and entropy compared with DPO. When applied to Llama-3-8B-Instruct, BPO achieves state-of-the-art performance among Llama-3-8B backbones, with a 55.9\% length-controlled win rate on AlpacaEval2. Project page: https://github.com/aailab-kaist/BPO.

Figures (10)

• Figure 1: The trade-off between fidelity and diversity across instances from different probabilistic preference optimization frameworks on dialogue generation with Pythia 2.8B.
• Figure 2: Point-wise Bregman divergence defined by $h$, between the model $R_\theta$ and the target $R_{\theta^*}$.
• Figure 3: Gradient magnitude and direction analysis across different Bregman divergences.
• Figure 4: Orthogonal improvements with BPO (SBA).
• Figure 5: Ablation studies on the effect of $\lambda$ in SBA, using Anthropic-HH with Pythia-2.8B.

Theorems & Definitions (12)

• Proposition 0
• Theorem 1
• Theorem 2
• Proposition 2
• Proposition 2
• proof
• Theorem 2
• proof
• Theorem 2
• proof