Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Greedy Sampling Is Provably Efficient for RLHF

Di Wu, Chengshuai Shi, Jing Yang, Cong Shen

TL;DR

The paper addresses RLHF under KL-regularized contextual bandits with preference feedback, extending theoretical guarantees to the general preference model and the BT special case. It introduces a Greedy Sampling algorithm that uses empirical estimates directly, exploiting KL-regularization to bound optimal policies relative to a reference policy. The authors prove online regret bounds of order $O( ext{exp}( olinebreak \eta) imes d(ullet) imes ext{log}(T))$ and offline sample complexities of $O(1/ olinebreak epsilon)$ for both GP and BT models, while avoiding confidence-bound constructions. Empirical results corroborate the theory, showing that greedy sampling achieves comparable performance to optimism-based methods with reduced computational overhead, suggesting practical efficiency for RLHF.

Abstract

Reinforcement Learning from Human Feedback (RLHF) has emerged as a key technique for post-training large language models. Despite its empirical success, the theoretical understanding of RLHF is still limited, as learning the KL-regularized target with only preference feedback poses additional challenges compared with canonical RL. Existing works mostly study the reward-based Bradley-Terry (BT) preference model, and extend classical designs utilizing optimism or pessimism. This work, instead, considers the general preference model (whose practical relevance has been observed recently) and obtains performance guarantees with major, order-wise improvements over existing ones. Surprisingly, these results are derived from algorithms that directly use the empirical estimates (i.e., greedy sampling), as opposed to constructing optimistic or pessimistic estimates in previous works. This insight has a deep root in the unique structural property of the optimal policy class under the KL-regularized target, and we further specialize it to the BT model, highlighting the surprising sufficiency of greedy sampling in RLHF.

Greedy Sampling Is Provably Efficient for RLHF

TL;DR

The paper addresses RLHF under KL-regularized contextual bandits with preference feedback, extending theoretical guarantees to the general preference model and the BT special case. It introduces a Greedy Sampling algorithm that uses empirical estimates directly, exploiting KL-regularization to bound optimal policies relative to a reference policy. The authors prove online regret bounds of order and offline sample complexities of for both GP and BT models, while avoiding confidence-bound constructions. Empirical results corroborate the theory, showing that greedy sampling achieves comparable performance to optimism-based methods with reduced computational overhead, suggesting practical efficiency for RLHF.

Abstract

Reinforcement Learning from Human Feedback (RLHF) has emerged as a key technique for post-training large language models. Despite its empirical success, the theoretical understanding of RLHF is still limited, as learning the KL-regularized target with only preference feedback poses additional challenges compared with canonical RL. Existing works mostly study the reward-based Bradley-Terry (BT) preference model, and extend classical designs utilizing optimism or pessimism. This work, instead, considers the general preference model (whose practical relevance has been observed recently) and obtains performance guarantees with major, order-wise improvements over existing ones. Surprisingly, these results are derived from algorithms that directly use the empirical estimates (i.e., greedy sampling), as opposed to constructing optimistic or pessimistic estimates in previous works. This insight has a deep root in the unique structural property of the optimal policy class under the KL-regularized target, and we further specialize it to the BT model, highlighting the surprising sufficiency of greedy sampling in RLHF.

Paper Structure

This paper contains 34 sections, 21 theorems, 98 equations, 4 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Problem Formulation and Preliminaries
  3. Properties of the Optimal Policy Class
  4. Greedy Sampling for Online Learning
  5. Algorithm Design
  6. Theoretical Analysis
  7. Greedy Sampling for Offline Learning
  8. Algorithm Design
  9. Theoretical Analysis
  10. Experiment
  11. Baselines
  12. Results
  13. Related Works
  14. Conclusion
  15. Discussions
  16. ...and 19 more sections

Key Result

Proposition 1

For any reward function $R^*$ and any preference model $P^*$, the corresponding optimal policy $\pi^*_\textup{BT}$ and the corresponding NE policy $\pi^*_\textup{GP}$ satisfy that where $\mathcal{F}:= \{f: f(x, a) \in [0, 1], \forall (x, a) \in \mathcal{X} \times \mathcal{A}\}$ denotes all the functions mapping context-action pairs to values in $[0, 1]$.

Figures (4)

  • Figure 1: The comparison between greedy sampling and optimism under the general preference model.
  • Figure 2: The comparison between greedy sampling and optimism under the Bradley-Terry model.
  • Figure 3: The comparison of different regularization coefficients $\eta$ under the general preference model.
  • Figure 4: Comparison of different regularization coefficients $\eta$ under the Bradley-Terry model.

Theorems & Definitions (28)

  • Definition 1: General Preference Model
  • Definition 2: Bradley-Terry Model
  • Proposition 1: Optimal Policy Class
  • Lemma 1
  • Definition 3: Eluder Dimension, General Preference Model
  • Theorem 1: Online, General Preference Model
  • Definition 4: Eluder Dimension, Bradley-Terry Model
  • Theorem 2: Online, Bradley-Terry Model
  • Definition 5: Data Coverage
  • Theorem 3: Offline, General Preference Model
  • ...and 18 more