Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Preference-Based Reinforcement Learning: Randomized Exploration Meets Experimental Design

Andreas Schlaginhaufen, Reda Ouhamma, Maryam Kamgarpour

TL;DR

This work tackles reinforcement learning from human preferences in general MDPs by introducing randomized exploration as a tractable alternative to optimistic methods. It presents two main meta-algorithms: RPO-Regret for online regret guarantees and RPO-Explore for batch, preference-free exploration, both leveraging an RL oracle and a learned reward parameter via maximum likelihood on trajectory-difference features. To improve practicality, it then proposes LRPO-OD-Regret, a lazy-update, design-based variant that enables parallel preference labeling and selective querying with D-optimal design, while retaining regret guarantees. Theoretical results show sublinear regret and favorable suboptimality bounds, and experiments in tabular and continuous control settings demonstrate competitive performance with significantly fewer preference queries. Overall, the paper advances efficient RLHF by connecting randomized exploration with experimental design to shrink human annotation burden while maintaining solid learning guarantees.

Abstract

We study reinforcement learning from human feedback in general Markov decision processes, where agents learn from trajectory-level preference comparisons. A central challenge in this setting is to design algorithms that select informative preference queries to identify the underlying reward while ensuring theoretical guarantees. We propose a meta-algorithm based on randomized exploration, which avoids the computational challenges associated with optimistic approaches and remains tractable. We establish both regret and last-iterate guarantees under mild reinforcement learning oracle assumptions. To improve query complexity, we introduce and analyze an improved algorithm that collects batches of trajectory pairs and applies optimal experimental design to select informative comparison queries. The batch structure also enables parallelization of preference queries, which is relevant in practical deployment as feedback can be gathered concurrently. Empirical evaluation confirms that the proposed method is competitive with reward-based reinforcement learning while requiring a small number of preference queries.

Efficient Preference-Based Reinforcement Learning: Randomized Exploration Meets Experimental Design

TL;DR

This work tackles reinforcement learning from human preferences in general MDPs by introducing randomized exploration as a tractable alternative to optimistic methods. It presents two main meta-algorithms: RPO-Regret for online regret guarantees and RPO-Explore for batch, preference-free exploration, both leveraging an RL oracle and a learned reward parameter via maximum likelihood on trajectory-difference features. To improve practicality, it then proposes LRPO-OD-Regret, a lazy-update, design-based variant that enables parallel preference labeling and selective querying with D-optimal design, while retaining regret guarantees. Theoretical results show sublinear regret and favorable suboptimality bounds, and experiments in tabular and continuous control settings demonstrate competitive performance with significantly fewer preference queries. Overall, the paper advances efficient RLHF by connecting randomized exploration with experimental design to shrink human annotation burden while maintaining solid learning guarantees.

Abstract

We study reinforcement learning from human feedback in general Markov decision processes, where agents learn from trajectory-level preference comparisons. A central challenge in this setting is to design algorithms that select informative preference queries to identify the underlying reward while ensuring theoretical guarantees. We propose a meta-algorithm based on randomized exploration, which avoids the computational challenges associated with optimistic approaches and remains tractable. We establish both regret and last-iterate guarantees under mild reinforcement learning oracle assumptions. To improve query complexity, we introduce and analyze an improved algorithm that collects batches of trajectory pairs and applies optimal experimental design to select informative comparison queries. The batch structure also enables parallelization of preference queries, which is relevant in practical deployment as feedback can be gathered concurrently. Empirical evaluation confirms that the proposed method is competitive with reward-based reinforcement learning while requiring a small number of preference queries.

Paper Structure

This paper contains 51 sections, 18 theorems, 73 equations, 5 figures, 4 algorithms.

Table of Contents

  1. Introduction
  2. Contributions
  3. Preliminaries
  4. Notation
  5. Setting
  6. Interaction protocol
  7. Preference model
  8. Regret
  9. Suboptimality
  10. Randomized Preference Optimization
  11. Algorithm
  12. Maximum likelihood estimation
  13. Randomized exploration
  14. RL oracle
  15. Algorithm statement
  16. ...and 36 more sections

Key Result

Lemma 3.0

Let $\lambda\geq 0$ and define the design matrix at time $t$ given by $V_t = \lambda I + \sum_{k=1}^{t-1}x_k x_k^\top$. Then, with probability $1-\delta$, for all $t\in\mathbb{N}$, the true reward parameter $\theta^*$ is contained in the ellipsoid Here, $\kappa:=\max_{\theta\in\mathcal{B}^d(B), x \in \mathcal{B}^d(2LH_{\gamma})}1/\dot{\sigma}\left({\langle \theta, x \rangle}\right)$ denotes the L

Figures (5)

  • Figure 1: We compare the regret of \ref{['alg:regret']} (orange, Algorithm \ref{['alg:regret']}) against a baseline with entropy exploration (blue). The solid lines indicate the median and the shaded areas the 0.2 and 0.8 quantiles, across 20 independent runs. The regret is computed with respect to the ground truth reward parameter $\theta^*$.
  • Figure 2: Comparison of regret minimization algorithms in terms of (a) the cumulative regret (estimated from samples against an RL policy trained with $\theta^*$) and (b) number of preference queries performed. In particular, we compare \ref{['alg:regret']} (orange, Algorithm \ref{['alg:regret']}) with its lazy versions \ref{['alg:lazy_thompson_sampling']} and \ref{['alg:lazy_thompson_sampling']} (blue & green). Here, \ref{['alg:lazy_thompson_sampling']} and \ref{['alg:lazy_thompson_sampling']} refer to Algorithm \ref{['alg:lazy_thompson_sampling']} without and with optimal design subroutine. The solid lines indicate the median and the shaded areas the 0.2 and 0.8 quantiles, across 20 independent runs.
  • Figure 3: Illustration of the gridworld environment.
  • Figure 4: Comparison of RLHF algorithms in terms of (a) the last iterate ground truth reward $V_{\theta^*}^{\hat{\pi}}$ (estimated from samples) and (b) number of preference queries performed. In particular, we compare RPO-Explore (green, Algorithm \ref{['alg:regret']}) with its lazy versions LRPO-Explore and LRPO-OD-Explore (orange & blue). Here, LRPO-Explore and LRPO-OD-Explore refer to Algorithm \ref{['alg:lazy_thompson_sampling']} without and with optimal design subroutine. The error bars indicate the 0.2 and 0.8 quantiles, across 10 independent runs. The dashed blue line indicates the mean reward achieved by PPO with the ground truth parameter $\theta^*$.
  • Figure 5: Comparing the ground truth rewards $V_{\theta^*}^{\pi_t}$ (estimated from samples) of RLHF algorithms for reward-free exploration during training, using the same color codes as in Figure \ref{['fig:experiments_pure_exploration']}.

Theorems & Definitions (32)

  • Remark 2.2
  • Lemma 3.0
  • Remark 3.1
  • Theorem 3.3
  • Theorem 3.4
  • Theorem 4.1
  • Theorem A.1
  • proof
  • Proposition A.1
  • proof
  • ...and 22 more