Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Online Bandit Learning with Offline Preference Data for Improved RLHF

Akhil Agnihotri, Rahul Jain, Deepak Ramachandran, Zheng Wen

TL;DR

This work addresses aligning generative models with human preferences by combining offline preference data with online reward-based fine-tuning in a RLHF-like setting. It formalizes the problem as a finite-armed linear bandit with an offline preference dataset and develops warmPref-PS, a Bayesian posterior sampling algorithm that warm-starts from the offline data and accounts for the rater’s competence. The authors establish information-theoretic conditions under which offline data is informative and prove a regret bound that degrades gracefully with offline data size and rater quality, showing sublinear regret as those resources grow. A practical Bootstrapped version is proposed to handle large or infinite arm spaces, and empirical results demonstrate substantial improvements over baselines and robustness to misspecification, indicating a viable path for RLHF enhancements using offline preferences.

Abstract

Reinforcement Learning with Human Feedback (RLHF) is at the core of fine-tuning methods for generative AI models for language and images. Such feedback is often sought as rank or preference feedback from human raters, as opposed to eliciting scores since the latter tends to be noisy. On the other hand, RL theory and algorithms predominantly assume that a reward feedback is available. In particular, approaches for online learning that can be helpful in adaptive data collection via active learning cannot incorporate offline preference data. In this paper, we adopt a finite-armed linear bandit model as a prototypical model of online learning. We consider an offline preference dataset to be available generated by an expert of unknown 'competence'. We propose warmPref-PS, a posterior sampling algorithm for online learning that can be warm-started with an offline dataset with noisy preference feedback. We show that by modeling the 'competence' of the expert that generated it, we are able to use such a dataset most effectively. We support our claims with novel theoretical analysis of its Bayesian regret, as well as, extensive empirical evaluation of an approximate loss function that optimizes for infinitely many arms, and performs substantially better than baselines.

Online Bandit Learning with Offline Preference Data for Improved RLHF

TL;DR

This work addresses aligning generative models with human preferences by combining offline preference data with online reward-based fine-tuning in a RLHF-like setting. It formalizes the problem as a finite-armed linear bandit with an offline preference dataset and develops warmPref-PS, a Bayesian posterior sampling algorithm that warm-starts from the offline data and accounts for the rater’s competence. The authors establish information-theoretic conditions under which offline data is informative and prove a regret bound that degrades gracefully with offline data size and rater quality, showing sublinear regret as those resources grow. A practical Bootstrapped version is proposed to handle large or infinite arm spaces, and empirical results demonstrate substantial improvements over baselines and robustness to misspecification, indicating a viable path for RLHF enhancements using offline preferences.

Abstract

Reinforcement Learning with Human Feedback (RLHF) is at the core of fine-tuning methods for generative AI models for language and images. Such feedback is often sought as rank or preference feedback from human raters, as opposed to eliciting scores since the latter tends to be noisy. On the other hand, RL theory and algorithms predominantly assume that a reward feedback is available. In particular, approaches for online learning that can be helpful in adaptive data collection via active learning cannot incorporate offline preference data. In this paper, we adopt a finite-armed linear bandit model as a prototypical model of online learning. We consider an offline preference dataset to be available generated by an expert of unknown 'competence'. We propose warmPref-PS, a posterior sampling algorithm for online learning that can be warm-started with an offline dataset with noisy preference feedback. We show that by modeling the 'competence' of the expert that generated it, we are able to use such a dataset most effectively. We support our claims with novel theoretical analysis of its Bayesian regret, as well as, extensive empirical evaluation of an approximate loss function that optimizes for infinitely many arms, and performs substantially better than baselines.
Paper Structure (35 sections, 13 theorems, 51 equations, 5 figures, 1 table, 2 algorithms)

This paper contains 35 sections, 13 theorems, 51 equations, 5 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Introducing the Preference-Warmed Posterior Sampling Algorithm
  4. Analysis of $\mathsf{warmPref-PS}$
  5. Informativeness of Offline Preference Data
  6. Regret Bound
  7. A Practical Approximation of the $\mathsf{warmPref-PS}$ Algorithm
  8. Empirical Results
  9. Conclusion
  10. Appendix
  11. Understanding Two Actions and Finite Deliberateness
  12. Constructing the Algorithm
  13. Limiting Behaviour of the Algorithm for Optimal Expert
  14. Limiting Behaviour of the Algorithm for Large Datasets
  15. Sample Complexity for Finite Deliberateness
  16. ...and 20 more sections

Key Result

Theorem 4.2

Let the action set ${\mathcal{A}}$ have size $K$ with a sampling distribution $\mu$ such that $0 < \mu_{\text{min}} \leq \mu_{k} \leq \mu_{\text{max}} < 1, \; \forall k \in [K]$. Given some $\epsilon \in (0, 1)$ and finite $\beta < \infty$, let $\lambda \to \infty$. Then, the singleton set ${\mathca and, $N$ is the size of the preference dataset and $\Phi(\cdot)$ is the standard Normal CDF.

Figures (5)

  • Figure 1: $\mathsf{warmPref-PS}$ (ours) comparison with IPO ipo and DPO rafailov2024direct. When divergence between $\pi_{\mathsf{ref}}$ and $\pi^{\star}$ is large, on-policy online learning is needed for best results.
  • Figure 2: Cumulative Regret comparison with varying $N$, $\beta$, and $\lambda$. Shaded region around the mean line represents 1 standard deviation over 5 independent runs.
  • Figure 3: Sensitivity analysis with flawed expert policy, misspecified and unknown competence.
  • Figure 4: Performance of warmTSOF with varying cost of feedback $c$.
  • Figure 5: Cumulative regret with varying $K$ and $T$.

Theorems & Definitions (28)

  • Remark 3.1
  • Definition 4.1
  • Theorem 4.2
  • Corollary 4.3
  • Theorem 4.4
  • Lemma 4.4
  • proof : Proof sketch
  • Theorem 4.5
  • Remark 4.6
  • Remark 4.7
  • ...and 18 more