Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Coverage Improvement and Fast Convergence of On-policy Preference Learning

Juno Kim, Jihun Yun, Jason D. Lee, Kwang-Sung Jun

TL;DR

The paper investigates why online on-policy preference learning can outperform offline approaches in language-model alignment. It introduces the coverage improvement principle, showing that on-policy updates progressively enter regions with better coverage, enabling rapid convergence; in a contextual-bandit setting with Bradley-Terry preferences and linear softmax policies, on-policy DPO achieves exponential convergence in the number of iterations once the batch size surpasses a generalized coverage threshold $C_*^p(R)$, yielding a total sample complexity of $O\left(\ln\frac{1}{\varepsilon}\,C_*^p(R)\right)$, in contrast to a minimax offline bound of $O\left(\frac{1}{\varepsilon^2}\,C_*^p(R)\right)$. The authors further propose a preferential G-optimal design to construct a two-round offline sampler that eliminates explicit coverage dependence and guarantees convergence in two rounds, and develop on-policy reward distillation schemes with improved, noiseless rates under a deviation-based coverage notion. Empirically, on-policy DPO and the reward-distillation methods outperform offline baselines and exhibit stable, monotonic gains across iterations, validating the theory and offering practical, scalable improvements for alignment tasks.

Abstract

Online on-policy preference learning algorithms for language model alignment such as online direct policy optimization (DPO) can significantly outperform their offline counterparts. We provide a theoretical explanation for this phenomenon by analyzing how the sampling policy's coverage evolves throughout on-policy training. We propose and rigorously justify the \emph{coverage improvement principle}: with sufficient batch size, each update moves into a region around the target where coverage is uniformly better, making subsequent data increasingly informative and enabling rapid convergence. In the contextual bandit setting with Bradley-Terry preferences and linear softmax policy class, we show that on-policy DPO converges exponentially in the number of iterations for batch size exceeding a generalized coverage threshold. In contrast, any learner restricted to offline samples from the initial policy suffers a slower minimax rate, leading to a sharp separation in total sample complexity. Motivated by this analysis, we further propose a simple hybrid sampler based on a novel \emph{preferential} G-optimal design, which removes dependence on coverage and guarantees convergence in just two rounds. Finally, we develop principled on-policy schemes for reward distillation in the general function class setting, and show faster noiseless rates under an alternative deviation-based notion of coverage. Experimentally, we confirm that on-policy DPO and our proposed reward distillation algorithms outperform their off-policy counterparts and enjoy stable, monotonic performance gains across iterations.

Coverage Improvement and Fast Convergence of On-policy Preference Learning

TL;DR

The paper investigates why online on-policy preference learning can outperform offline approaches in language-model alignment. It introduces the coverage improvement principle, showing that on-policy updates progressively enter regions with better coverage, enabling rapid convergence; in a contextual-bandit setting with Bradley-Terry preferences and linear softmax policies, on-policy DPO achieves exponential convergence in the number of iterations once the batch size surpasses a generalized coverage threshold , yielding a total sample complexity of , in contrast to a minimax offline bound of . The authors further propose a preferential G-optimal design to construct a two-round offline sampler that eliminates explicit coverage dependence and guarantees convergence in two rounds, and develop on-policy reward distillation schemes with improved, noiseless rates under a deviation-based coverage notion. Empirically, on-policy DPO and the reward-distillation methods outperform offline baselines and exhibit stable, monotonic gains across iterations, validating the theory and offering practical, scalable improvements for alignment tasks.

Abstract

Online on-policy preference learning algorithms for language model alignment such as online direct policy optimization (DPO) can significantly outperform their offline counterparts. We provide a theoretical explanation for this phenomenon by analyzing how the sampling policy's coverage evolves throughout on-policy training. We propose and rigorously justify the \emph{coverage improvement principle}: with sufficient batch size, each update moves into a region around the target where coverage is uniformly better, making subsequent data increasingly informative and enabling rapid convergence. In the contextual bandit setting with Bradley-Terry preferences and linear softmax policy class, we show that on-policy DPO converges exponentially in the number of iterations for batch size exceeding a generalized coverage threshold. In contrast, any learner restricted to offline samples from the initial policy suffers a slower minimax rate, leading to a sharp separation in total sample complexity. Motivated by this analysis, we further propose a simple hybrid sampler based on a novel \emph{preferential} G-optimal design, which removes dependence on coverage and guarantees convergence in just two rounds. Finally, we develop principled on-policy schemes for reward distillation in the general function class setting, and show faster noiseless rates under an alternative deviation-based notion of coverage. Experimentally, we confirm that on-policy DPO and our proposed reward distillation algorithms outperform their off-policy counterparts and enjoy stable, monotonic performance gains across iterations.
Paper Structure (57 sections, 25 theorems, 171 equations, 2 figures, 2 tables, 3 algorithms)

This paper contains 57 sections, 25 theorems, 171 equations, 2 figures, 2 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Our contributions.
  3. Formalization of Alignment
  4. Problem Setup
  5. Notation.
  6. Learning from preferences.
  7. Direct preference optimization (DPO).
  8. Policy class.
  9. Generalized coverage.
  10. Minimax Lower Bound for Offline Preference Learning
  11. Fast Convergence of On-policy DPO
  12. On-policy DPO.
  13. Convergence Analysis
  14. Faster convergence under super-quadraticity.
  15. Two-Step Convergence with Preferential Optimal Design
  16. ...and 42 more sections

Key Result

Proposition 2.1

Let $\pi_\theta = \mathop{\mathrm{softmax}}\nolimits(\theta)$ for $\theta\in\mathbb{R}^d$. Then if $\lVert\theta^*\rVert_\infty = O(1)$ and $d=O(\operatorname{poly}(R))$, it holds that $\mathbb{V}(\pi^*)|_S \gtrsim \frac{1}{d}{\mathbf{I}}_S$ and $C_*^p(R)\asymp e^R$ when $p=1$, or $C_*^p(R)\asymp \f

Figures (2)

  • Figure 1: A sequence of iteratively shrinking confidence regions $B_p(\htheta_k,r_k)$ and target $\theta^*$.
  • Figure 2: Win-rate (left) and reward score (right) of on-policy versus off-policy DPO over $5$ iterations. Curves represent the mean performance over 5 runs with standard deviations indicated.

Theorems & Definitions (41)

  • Proposition 2.1
  • Theorem 2.2: Lower bound for offline preference learning
  • Proposition 3.1: One-step policy improvement
  • Theorem 3.2: Fast convergence of on-policy DPO
  • Corollary 3.3
  • Remark 3.4
  • Remark 3.5
  • Corollary 3.6
  • Lemma 4.1
  • Proposition 4.2: Preferential G-optimal design
  • ...and 31 more