Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Direct Density Ratio Optimization: A Statistically Consistent Approach to Aligning Large Language Models

Rei Higuchi, Taiji Suzuki

TL;DR

Direct Density Ratio Optimization (DDRO) offers a statistically consistent framework for aligning LLMs by directly estimating the density ratio between preferred and unpreferred outputs, circumventing explicit human-preference models. By formulating the problem via a Bregman-divergence loss on the density ratio and leveraging unpaired data, DDRO provably converges to the true preferred distribution $p^+$ as data grows. Theoretical guarantees are complemented by strong empirical results on UF-G and UF-B benchmarks across multiple model sizes, often matching or surpassing state-of-the-art methods while using unpaired data. This data-driven, model-agnostic approach significantly enhances the reliability and scalability of LLM alignment in real-world settings.

Abstract

Aligning large language models (LLMs) with human preferences is crucial for safe deployment, yet existing methods assume specific preference models like Bradley-Terry model. This assumption leads to statistical inconsistency, where more data doesn't guarantee convergence to true human preferences. To address this critical gap, we introduce a novel alignment method Direct Density Ratio Optimization (DDRO). DDRO directly estimates the density ratio between preferred and unpreferred output distributions, circumventing the need for explicit human preference modeling. We theoretically prove that DDRO is statistically consistent, ensuring convergence to the true preferred distribution as the data size grows, regardless of the underlying preference structure. Experiments demonstrate that DDRO achieves superior performance compared to existing methods on many major benchmarks. DDRO unlocks the potential for truly data-driven alignment, paving the way for more reliable and human-aligned LLMs.

Direct Density Ratio Optimization: A Statistically Consistent Approach to Aligning Large Language Models

TL;DR

Direct Density Ratio Optimization (DDRO) offers a statistically consistent framework for aligning LLMs by directly estimating the density ratio between preferred and unpreferred outputs, circumventing explicit human-preference models. By formulating the problem via a Bregman-divergence loss on the density ratio and leveraging unpaired data, DDRO provably converges to the true preferred distribution as data grows. Theoretical guarantees are complemented by strong empirical results on UF-G and UF-B benchmarks across multiple model sizes, often matching or surpassing state-of-the-art methods while using unpaired data. This data-driven, model-agnostic approach significantly enhances the reliability and scalability of LLM alignment in real-world settings.

Abstract

Aligning large language models (LLMs) with human preferences is crucial for safe deployment, yet existing methods assume specific preference models like Bradley-Terry model. This assumption leads to statistical inconsistency, where more data doesn't guarantee convergence to true human preferences. To address this critical gap, we introduce a novel alignment method Direct Density Ratio Optimization (DDRO). DDRO directly estimates the density ratio between preferred and unpreferred output distributions, circumventing the need for explicit human preference modeling. We theoretically prove that DDRO is statistically consistent, ensuring convergence to the true preferred distribution as the data size grows, regardless of the underlying preference structure. Experiments demonstrate that DDRO achieves superior performance compared to existing methods on many major benchmarks. DDRO unlocks the potential for truly data-driven alignment, paving the way for more reliable and human-aligned LLMs.
Paper Structure (25 sections, 4 theorems, 39 equations, 3 figures, 3 tables)

This paper contains 25 sections, 4 theorems, 39 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work: Limitations of Existing Alignment Methods
  3. Direct Density Ratio Optimization (DDRO)
  4. Problem Setting
  5. DDRO: The Proposed Method
  6. Empirical estimate of the Bregman divergence loss.
  7. Related work on density ratio estimation
  8. Practical modification of DDRO
  9. Theoretical Analysis
  10. Estimation error bound
  11. Connection to existing methods
  12. Experiments
  13. Experimental Setup
  14. Datasets.
  15. Benchmarks.
  16. ...and 10 more sections

Key Result

Proposition 2.1

There exists a class of preferences, none of which can be obtained by minimizing the negative log-likelihood loss under the Bradley-Terry model assumption.

Figures (3)

  • Figure 1: Performance comparison of three methods (KTO, BCO, DDRO) on various benchmarks (BBH, GSM8K, MMLU, TruthfulQA, and AlpacaEval LC Winrate). The methods are applied to four different model sizes: Pythia 1.4B, Pythia 2.8B, Pythia 6.9B, and LLaMA 7B, each using the UF-G dataset.
  • Figure 2: Performance comparison of DDRO and DPO on various benchmarks (BBH, GSM8K, MMLU, TruthfulQA, and AlpacaEval LC Winrate). The methods are applied to four different model sizes: Pythia 1.4B, Pythia 2.8B, Pythia 6.9B, and LLaMA 7B, each using the UF-B dataset.
  • Figure 3: Gradient norm during training for different smoothing functions $S(x)$ (identity: $S(x) = x$, logsig: $S(x) = \log \sigma(x)$, neglogsigneg: $S(x) = - \log \sigma(-x)$, sig: $S(x) = \sigma(x)$).

Theorems & Definitions (9)

  • Proposition 2.1
  • Proposition 3.3
  • Example 3.4
  • Theorem 4.1
  • proof
  • proof
  • proof
  • Lemma 3.1
  • proof