Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Preference-based Reinforcement Learning beyond Pairwise Comparisons: Benefits of Multiple Options

Joongkyu Lee, Seouh-won Yi, Min-hwan Oh

TL;DR

This paper advances online preference-based reinforcement learning by modeling ranking feedback over action subsets via the Plackett–Luce model and proposing M-AUPO, which greedily selects assortments to maximize average uncertainty. The key theoretical contributions show that the suboptimality gap scales as $ ilde{O}ig( rac{d}{T}ig( extstyleig( rac{1}{|S_t|}ig)^{1/2}ig)ig)$, with an additional term that vanishes for large $T$, and crucially removes the customary $e^{B}$ dependence; a near-matching lower bound $oldsymbol{Omega}ig( rac{d}{K\, ext{sqrt}(T)}ig)$ confirms the benefit of larger subsets. The RB variant provides computational and empirical advantages, and a warm-up phase technique helps bound the leading term without auxiliary tricks. Experiments on synthetic data and real LLM-related datasets (TREC-DL and NECTAR) corroborate that increasing $K$ improves sample efficiency and that RB-based estimation yields strong practical performance. Overall, the work significantly clarifies how richer ranking feedback can tighten PbRL guarantees and informs scalable design for ranking-based learning-to-rank in contextual settings.

Abstract

We study online preference-based reinforcement learning (PbRL) with the goal of improving sample efficiency. While a growing body of theoretical work has emerged-motivated by PbRL's recent empirical success, particularly in aligning large language models (LLMs)-most existing studies focus only on pairwise comparisons. A few recent works (Zhu et al., 2023, Mukherjee et al., 2024, Thekumparampil et al., 2024) have explored using multiple comparisons and ranking feedback, but their performance guarantees fail to improve-and can even deteriorate-as the feedback length increases, despite the richer information available. To address this gap, we adopt the Plackett-Luce (PL) model for ranking feedback over action subsets and propose M-AUPO, an algorithm that selects multiple actions by maximizing the average uncertainty within the offered subset. We prove that M-AUPO achieves a suboptimality gap of $\tilde{O}\left( \frac{d}{T} \sqrt{ \sum_{t=1}^T \frac{1}{|S_t|}} \right)$, where $T$ is the total number of rounds, $d$ is the feature dimension, and $|S_t|$ is the size of the subset at round $t$. This result shows that larger subsets directly lead to improved performance and, notably, the bound avoids the exponential dependence on the unknown parameter's norm, which was a fundamental limitation in most previous works. Moreover, we establish a near-matching lower bound of $Ω\left( \frac{d}{K \sqrt{T}} \right)$, where $K$ is the maximum subset size. To the best of our knowledge, this is the first theoretical result in PbRL with ranking feedback that explicitly shows improved sample efficiency as a function of the subset size.

Preference-based Reinforcement Learning beyond Pairwise Comparisons: Benefits of Multiple Options

TL;DR

This paper advances online preference-based reinforcement learning by modeling ranking feedback over action subsets via the Plackett–Luce model and proposing M-AUPO, which greedily selects assortments to maximize average uncertainty. The key theoretical contributions show that the suboptimality gap scales as , with an additional term that vanishes for large , and crucially removes the customary dependence; a near-matching lower bound confirms the benefit of larger subsets. The RB variant provides computational and empirical advantages, and a warm-up phase technique helps bound the leading term without auxiliary tricks. Experiments on synthetic data and real LLM-related datasets (TREC-DL and NECTAR) corroborate that increasing improves sample efficiency and that RB-based estimation yields strong practical performance. Overall, the work significantly clarifies how richer ranking feedback can tighten PbRL guarantees and informs scalable design for ranking-based learning-to-rank in contextual settings.

Abstract

We study online preference-based reinforcement learning (PbRL) with the goal of improving sample efficiency. While a growing body of theoretical work has emerged-motivated by PbRL's recent empirical success, particularly in aligning large language models (LLMs)-most existing studies focus only on pairwise comparisons. A few recent works (Zhu et al., 2023, Mukherjee et al., 2024, Thekumparampil et al., 2024) have explored using multiple comparisons and ranking feedback, but their performance guarantees fail to improve-and can even deteriorate-as the feedback length increases, despite the richer information available. To address this gap, we adopt the Plackett-Luce (PL) model for ranking feedback over action subsets and propose M-AUPO, an algorithm that selects multiple actions by maximizing the average uncertainty within the offered subset. We prove that M-AUPO achieves a suboptimality gap of , where is the total number of rounds, is the feature dimension, and is the size of the subset at round . This result shows that larger subsets directly lead to improved performance and, notably, the bound avoids the exponential dependence on the unknown parameter's norm, which was a fundamental limitation in most previous works. Moreover, we establish a near-matching lower bound of , where is the maximum subset size. To the best of our knowledge, this is the first theoretical result in PbRL with ranking feedback that explicitly shows improved sample efficiency as a function of the subset size.
Paper Structure (38 sections, 23 theorems, 154 equations, 8 figures, 5 tables, 3 algorithms)

This paper contains 38 sections, 23 theorems, 154 equations, 8 figures, 5 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. Problem Setting and Preliminaries
  4. Loss Functions and Rank-Breaking
  5. Online Parameter Estimation
  6. M-AUPO: Maximizing Average Uncertainty
  7. Main Results
  8. Suboptimality Gap of M-AUPO
  9. Lower Bound
  10. Numerical Experiments
  11. Synthetic Data
  12. Real-World Dataset
  13. Conclusion
  14. Appendix
  15. Further Related Work
  16. ...and 23 more sections

Key Result

Corollary 1

Let $\delta \in (0, 1]$. We set $\eta = (1+ 3\sqrt{2} B)/2$ and $\lambda = \max \{ 12 \sqrt{2} B \eta, 144 \eta d, 2 \}$. Then, under Assumption assum:linear_reward, with probability at least $1 - \delta$, we have where $H_t^{(j)} := H_t + \sum_{j'=1}^{j-1} \nabla^2 \ell_s^{(j')} (\widehat{\bm{\theta}}_{s}^{(j'+1)}) + \lambda \mathbf{I}_d$.

Figures (8)

  • Figure 1: Synthetic data experiment: suboptimality gap of M-AUPO under varying $K$, evaluated with PL loss (left) and RB loss (middle), along with comparison against other baselines (right).
  • Figure 2: Real-world dataset experiment: suboptimality gap of M-AUPO under varying $K$ on the TREC-DL dataset (left) and the NECTAR dataset (middle), along with comparison against other baselines (right). The results are rescaled to align the performances between the two datasets.
  • Figure : OMD-PL, OMD for PL Loss
  • Figure G.1: Performance comparisons for Instance 1 (Stochastic contexts) with $K = 2$, $3$, and $5$, evaluated under the PL loss (first row) and RB loss (second row).
  • Figure H.1: Performance comparisons for Instance 2 (Non-contextual) with $K = 2$, $3$, and $5$, evaluated under the PL loss (first row) and RB loss (second row).
  • ...and 3 more figures

Theorems & Definitions (26)

  • Definition 1: PL model
  • Remark 1: Computational cost of OMD
  • Corollary 1: Online confidence bound for PL loss
  • Theorem 1: Suboptimality gap for PL loss
  • Remark 2: Generality of our technique for avoiding $\mathcal{O}(e^B)$
  • Theorem 2: Online confidence bound for RB loss
  • Theorem 3: Lower bound
  • Lemma C.1: Online confidence bound, Theorem 4.2 of lee2025improved
  • Corollary C.1: Restatement of Corollary \ref{['cor:online_CB_PL_main']}, Online confidence bound for PL loss
  • Lemma C.2: Empirical-to-expected Hessian lower bound
  • ...and 16 more