Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models

Tarun Raheja, Nilay Pochhi

TL;DR

The paper tackles the problem of how to align large language models with human preferences by unifying RLHF and its numerous alternatives under a single theoretical framework, PsiPO, organized along three axes: Preference Model, Regularization, and Data Distribution. It derives formal results, including a coverage separation between online and offline methods and scaling laws for reward overoptimization, and clarifies failure modes such as length hacking and likelihood displacement. The work synthesizes over 50 papers into a coherent taxonomy, provides a practical decision guide (when to use PPO, DPO, SimPO, etc.), and highlights open problems like richer feedback and multi-objective alignment. The significance lies in enabling principled method selection and robust, scalable alignment of LLMs as models grow in capability and deployment scope.

Abstract

Aligning large language models (LLMs) with human preferences has become essential for safe and beneficial AI deployment. While Reinforcement Learning from Human Feedback (RLHF) established the dominant paradigm, a proliferation of alternatives -- Direct Preference Optimization (DPO), Identity Preference Optimization (IPO), Kahneman-Tversky Optimization (KTO), Simple Preference Optimization (SimPO), and many others -- has left practitioners without clear guidance on method selection. This survey provides a \textit{theoretical unification} of preference learning methods, revealing that the apparent diversity reduces to principled choices along three orthogonal axes: \textbf{(I) Preference Model} (what likelihood model underlies the objective), \textbf{(II) Regularization Mechanism} (how deviation from reference policies is controlled), and \textbf{(III) Data Distribution} (online vs.\ offline learning and coverage requirements). We formalize each axis with precise definitions and theorems, establishing key results including the coverage separation between online and offline methods, scaling laws for reward overoptimization, and conditions under which direct alignment methods fail. Our analysis reveals that failure modes -- length hacking, mode collapse, likelihood displacement -- arise from specific, predictable combinations of design choices. We synthesize empirical findings across 50+ papers and provide a practitioner's decision guide for method selection. The framework transforms preference learning from an empirical art into a theoretically grounded discipline.

From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models

TL;DR

The paper tackles the problem of how to align large language models with human preferences by unifying RLHF and its numerous alternatives under a single theoretical framework, PsiPO, organized along three axes: Preference Model, Regularization, and Data Distribution. It derives formal results, including a coverage separation between online and offline methods and scaling laws for reward overoptimization, and clarifies failure modes such as length hacking and likelihood displacement. The work synthesizes over 50 papers into a coherent taxonomy, provides a practical decision guide (when to use PPO, DPO, SimPO, etc.), and highlights open problems like richer feedback and multi-objective alignment. The significance lies in enabling principled method selection and robust, scalable alignment of LLMs as models grow in capability and deployment scope.

Abstract

Aligning large language models (LLMs) with human preferences has become essential for safe and beneficial AI deployment. While Reinforcement Learning from Human Feedback (RLHF) established the dominant paradigm, a proliferation of alternatives -- Direct Preference Optimization (DPO), Identity Preference Optimization (IPO), Kahneman-Tversky Optimization (KTO), Simple Preference Optimization (SimPO), and many others -- has left practitioners without clear guidance on method selection. This survey provides a \textit{theoretical unification} of preference learning methods, revealing that the apparent diversity reduces to principled choices along three orthogonal axes: \textbf{(I) Preference Model} (what likelihood model underlies the objective), \textbf{(II) Regularization Mechanism} (how deviation from reference policies is controlled), and \textbf{(III) Data Distribution} (online vs.\ offline learning and coverage requirements). We formalize each axis with precise definitions and theorems, establishing key results including the coverage separation between online and offline methods, scaling laws for reward overoptimization, and conditions under which direct alignment methods fail. Our analysis reveals that failure modes -- length hacking, mode collapse, likelihood displacement -- arise from specific, predictable combinations of design choices. We synthesize empirical findings across 50+ papers and provide a practitioner's decision guide for method selection. The framework transforms preference learning from an empirical art into a theoretically grounded discipline.
Paper Structure (57 sections, 12 theorems, 25 equations, 1 figure, 4 tables)

This paper contains 57 sections, 12 theorems, 25 equations, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. The Orthodoxy.
  3. The Challenge.
  4. The Resolution.
  5. Contributions.
  6. Preliminaries: Choice Theory Meets Language Models
  7. Notation and Setup
  8. The Bradley-Terry Model
  9. The Canonical RLHF Objective
  10. A Unified Framework: $\Psi$PO
  11. The Two Approximations in RLHF
  12. The General $\Psi$PO Objective
  13. DPO: The Reparameterization Trick
  14. Beyond DPO: IPO, KTO, and Reference-Free Methods
  15. IPO.
  16. ...and 42 more sections

Key Result

Theorem 2.3

The optimal policy for eq:rlhf has the closed form: where $Z(x) = \sum_y \pi_{\mathrm{ref}}(y|x) \exp(r(x,y)/\beta)$ is the partition function.

Figures (1)

  • Figure 1: Reward Overoptimization. Proxy reward increases monotonically with KL from reference, while true reward peaks then declines.

Theorems & Definitions (20)

  • Definition 2.1: Bradley-Terry Preference Model
  • Definition 2.2: KL-Regularized Reward Maximization
  • Theorem 2.3: Optimal Policy Form rafailov2023directpeters2010relative
  • Definition 3.1: $\Psi$PO Objective azar2023general
  • Theorem 3.2: Instantiations of $\Psi$PO
  • Theorem 3.3: DPO Reparameterization rafailov2023direct
  • Proposition 4.2: Identification Failure qin2024dpoheterogeneous
  • Definition 4.3: Nash Learning from Human Feedback munos2024nash
  • Proposition 5.1: DPO's Implicit Regularization azar2023general
  • Theorem 5.2: Preference Collapse xiao2024algorithmic
  • ...and 10 more