Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Preference is More Than Comparisons: Rethinking Dueling Bandits with Augmented Human Feedback

Shengbo Wang, Hong Sun, Ke Li

TL;DR

This work revisits dueling bandits for interactive preference elicitation by introducing augmented human feedback into a model-free framework. The key idea is to augment confidence bounds with related observations derived from contextual dependencies, yielding a generalized concentration property and a calibration threshold that guide when augmentation helps. The authors develop IPEA-RUCB and IPEA-DTS, providing regret and sample-complexity analyses that reveal a trade-off governed by dependency strength and augmentation weight. Empirically, the approach demonstrates competitive sample efficiency across item recommendation, multi-objective optimization, and LLM response tasks, and the framework is readily extensible to richer feedback modalities. Overall, the paper offers a principled, extensible foundation for efficient IPE in diverse personalization contexts by unifying augmented feedback with DB theory and providing concrete algorithmic designs and theoretical guarantees.

Abstract

Interactive preference elicitation (IPE) aims to substantially reduce human effort while acquiring human preferences in wide personalization systems. Dueling bandit (DB) algorithms enable optimal decision-making in IPE building on pairwise comparisons. However, they remain inefficient when human feedback is sparse. Existing methods address sparsity by heavily relying on parametric reward models, whose rigid assumptions are vulnerable to misspecification. In contrast, we explore an alternative perspective based on feedback augmentation, and introduce critical improvements to the model-free DB framework. Specifically, we introduce augmented confidence bounds to integrate augmented human feedback under generalized concentration properties, and analyze the multi-factored performance trade-off via regret analysis. Our prototype algorithm achieves competitive performance across several IPE benchmarks, including recommendation, multi-objective optimization, and response optimization for large language models, demonstrating the potential of our approach for provably efficient IPE in broader applications.

Preference is More Than Comparisons: Rethinking Dueling Bandits with Augmented Human Feedback

TL;DR

This work revisits dueling bandits for interactive preference elicitation by introducing augmented human feedback into a model-free framework. The key idea is to augment confidence bounds with related observations derived from contextual dependencies, yielding a generalized concentration property and a calibration threshold that guide when augmentation helps. The authors develop IPEA-RUCB and IPEA-DTS, providing regret and sample-complexity analyses that reveal a trade-off governed by dependency strength and augmentation weight. Empirically, the approach demonstrates competitive sample efficiency across item recommendation, multi-objective optimization, and LLM response tasks, and the framework is readily extensible to richer feedback modalities. Overall, the paper offers a principled, extensible foundation for efficient IPE in diverse personalization contexts by unifying augmented feedback with DB theory and providing concrete algorithmic designs and theoretical guarantees.

Abstract

Interactive preference elicitation (IPE) aims to substantially reduce human effort while acquiring human preferences in wide personalization systems. Dueling bandit (DB) algorithms enable optimal decision-making in IPE building on pairwise comparisons. However, they remain inefficient when human feedback is sparse. Existing methods address sparsity by heavily relying on parametric reward models, whose rigid assumptions are vulnerable to misspecification. In contrast, we explore an alternative perspective based on feedback augmentation, and introduce critical improvements to the model-free DB framework. Specifically, we introduce augmented confidence bounds to integrate augmented human feedback under generalized concentration properties, and analyze the multi-factored performance trade-off via regret analysis. Our prototype algorithm achieves competitive performance across several IPE benchmarks, including recommendation, multi-objective optimization, and response optimization for large language models, demonstrating the potential of our approach for provably efficient IPE in broader applications.

Paper Structure

This paper contains 49 sections, 4 theorems, 27 equations, 15 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Method
  4. AugConfidanceBound: Integrating Confidence Bounds with Augmented Feedback
  5. Concentration property
  6. Calibration threshold
  7. Connections to other DB approaches
  8. DuelingBanditAlgo: Decision-Making with Augmented Feedback
  9. Sample Efficiency
  10. Regret Analysis
  11. DependencyExtract & FeedbackAug: Computational Design
  12. Richer Forms of Human Feedback
  13. Empirical Study
  14. Item Recommendation
  15. Multi-Objective Optimization
  16. ...and 34 more sections

Key Result

Theorem 3.1

Assume $X^k_{i,j} \sim \mathrm{Bernoulli}( w^k_{i,j} p_{i,j})$ and let $C(\delta) = \left(\frac{(4 \alpha - 1) K^2}{(2 \alpha - 1) \delta}\right)^{\frac{1}{2\alpha - 1}}$. Given the preference matrix $\mathbf{P}$ with $K$ arms, then, for any $\alpha > 0.5$ and $\delta \in (0, 1)$, we have:

Figures (15)

  • Figure 1: Comparison of three DB approaches: context-free DB, structured reward estimation, and DB with augmented feedback.
  • Figure 2: Comparison of confidence intervals adding a direct observation or a related observation. The curve shows the derivative of the bound in equation \ref{['eqn:thm1-concentration']}, where the shaded area is the probability that the confidence interval fails to contain $p_{i,j}$.
  • Figure 3: Illustrative cases for bidirectional dependency. (a) Candidate partitioning. (b) Dependent arms with symmetric correlations. (c) General case without bidirectional dependency. (d) Two-objective trade-off solutions whose dependencies are determined by their distances. Given $[a,b,c,d,e]$, the candidates can be safely grouped into two subsets. When $c$ shifts to $c^\prime$, the dependencies weaken, making soft clustering a more suitable choice.
  • Figure 4: Illustration of computational design of dependency extraction and feedback augmentation.
  • Figure 5: Regret trajectories on Sushi and Car Preference.
  • ...and 10 more figures

Theorems & Definitions (7)

  • Theorem 3.1
  • Theorem 3.2
  • Theorem 3.3
  • Lemma 11.1: Chernoff-Hoeffding inequality
  • proof
  • proof
  • proof