Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

$f$-PO: Generalizing Preference Optimization with $f$-divergence Minimization

Jiaqi Han, Mingjian Jiang, Yuxuan Song, Stefano Ermon, Minkai Xu

TL;DR

This work introduces $f$-PO, a distribution-matching framework for language-model alignment that minimizes $\mathbb{D}_f(\hat{\pi}_\theta\|\hat{\pi}^*)$ between the parameterized policy and the optimal policy. By defining $\hat{\pi}_\theta(y|x) \propto \pi_\theta(y|x)^{\beta} \pi_{ref}(y|x)^{1-\beta}$ and $\hat{\pi}^* \propto \pi_{ref}(y|x) \exp(r(x,y))$, the approach generalizes and unifies existing offline preference methods such as DPO (reverse KL) and EXO (forward KL) and enables new divergences like $\alpha$-divergence. The authors provide theoretical results linking the $f$-PO objective to $f$-divergence between distributions, derive practical objectives for pairwise preference data, and perform extensive experiments showing that $\alpha$-PO often yields state-of-the-art or competitive performance on benchmarks including AlpacaEval 2, Arena-Hard, MT-Bench, and Open LLM Leaderboard v2, with ablations highlighting the effects of divergence choice. They also propose empirical refinements that improve training efficiency by reducing reliance on the reference model. Overall, $f$-PO offers a principled, flexible, and empirically effective framework for offline preference optimization and LM alignment.

Abstract

Preference optimization has made significant progress recently, with numerous methods developed to align language models with human preferences. This paper introduces $f$-divergence Preference Optimization ($f$-PO), a novel framework that generalizes and extends existing approaches. $f$-PO minimizes $f$-divergences between the optimized policy and the optimal policy, encompassing a broad family of alignment methods using various divergences. Our approach unifies previous algorithms like DPO and EXO, while offering new variants through different choices of $f$-divergences. We provide theoretical analysis of $f$-PO's properties and conduct extensive experiments on state-of-the-art language models using benchmark datasets. Results demonstrate $f$-PO's effectiveness across various tasks, achieving superior performance compared to existing methods on popular benchmarks such as AlpacaEval 2, Arena-Hard, MT-Bench, and Open LLM Leaderboard v2. Additionally, we present ablation studies exploring the impact of different $f$-divergences, offering insights into the trade-offs between regularization and performance in offline preference optimization. Our work contributes both practical algorithms and theoretical understanding to the field of language model alignment. Code is available at https://github.com/MinkaiXu/fPO.

$f$-PO: Generalizing Preference Optimization with $f$-divergence Minimization

TL;DR

This work introduces -PO, a distribution-matching framework for language-model alignment that minimizes between the parameterized policy and the optimal policy. By defining and , the approach generalizes and unifies existing offline preference methods such as DPO (reverse KL) and EXO (forward KL) and enables new divergences like -divergence. The authors provide theoretical results linking the -PO objective to -divergence between distributions, derive practical objectives for pairwise preference data, and perform extensive experiments showing that -PO often yields state-of-the-art or competitive performance on benchmarks including AlpacaEval 2, Arena-Hard, MT-Bench, and Open LLM Leaderboard v2, with ablations highlighting the effects of divergence choice. They also propose empirical refinements that improve training efficiency by reducing reliance on the reference model. Overall, -PO offers a principled, flexible, and empirically effective framework for offline preference optimization and LM alignment.

Abstract

Preference optimization has made significant progress recently, with numerous methods developed to align language models with human preferences. This paper introduces -divergence Preference Optimization (-PO), a novel framework that generalizes and extends existing approaches. -PO minimizes -divergences between the optimized policy and the optimal policy, encompassing a broad family of alignment methods using various divergences. Our approach unifies previous algorithms like DPO and EXO, while offering new variants through different choices of -divergences. We provide theoretical analysis of -PO's properties and conduct extensive experiments on state-of-the-art language models using benchmark datasets. Results demonstrate -PO's effectiveness across various tasks, achieving superior performance compared to existing methods on popular benchmarks such as AlpacaEval 2, Arena-Hard, MT-Bench, and Open LLM Leaderboard v2. Additionally, we present ablation studies exploring the impact of different -divergences, offering insights into the trade-offs between regularization and performance in offline preference optimization. Our work contributes both practical algorithms and theoretical understanding to the field of language model alignment. Code is available at https://github.com/MinkaiXu/fPO.

Paper Structure

This paper contains 26 sections, 5 theorems, 28 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Reinforcement Learning from Human Feedback
  5. Preference Optimization
  6. $f$-divergence
  7. Methodology
  8. The $f$-PO Framework
  9. Generalized Preference Optimization
  10. Instances of $f$-PO
  11. Empirical Variants with Approximations
  12. Experiments
  13. Experiments on the Choice of $f$ in $f$-PO
  14. Benchmarking $f$-PO
  15. Conclusion
  16. ...and 11 more sections

Key Result

Theorem 1

Let $\hat{\pi}_\theta(y|x) \propto \pi_\theta(y|x)^{\beta} \pi_{\textnormal{ref}}(y|x)^{(1-{\beta})}$ and $\hat{\pi}^* \propto \pi_{\textnormal{ref}} \exp{\left(r(x,y)\right)}$. We define our alignment objective as minimizing the following $f$-divergence: With unlimited model capacity and perfect optimization, the optimal policy $\pi^*_\theta$ satisfies that $\pi^*_\theta = \pi^*$.

Figures (3)

  • Figure 1: Illustration of the behavior of several divergences $\mathbb{D}_f(\pi_\theta\| \pi^*)$ in the $f$-divergence family. $\pi^\ast$ indicates the target policy while $\pi_\theta$ is the parameterized policy obtained by optimizing $\theta$ through minimizing certain $f$-divergence. Notably, $\alpha$-divergence exhibits varying characteristics when adjusting the values of $\alpha$.
  • Figure 2: The win rate (%) against SFT and Chosen of Pythia-2.8B fine-tuned by $f$-PO on Anthropic HH dataset with different values of $\alpha$.
  • Figure 3: The win rate (%) of Pythia-2.8B fine-tuned by 6 different $f$-divergences using $f$-PO on Anthropic HH dataset. $\alpha$-divergence generally performs the best.

Theorems & Definitions (9)

  • Theorem 1
  • Theorem 2
  • proof
  • Theorem 1
  • Lemma 1
  • proof
  • Theorem 2
  • proof
  • proof