Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Feel-Good Thompson Sampling for Contextual Dueling Bandits

Xuheng Li, Heyang Zhao, Quanquan Gu

TL;DR

This work introduces FGTS.CDB, a Feel-Good Thompson sampling algorithm for linear contextual dueling bandits, addressing the absence of posterior-sampling methods in this setting. By integrating a Feel-Good exploration term and exploiting independence between the two arms, FGTS.CDB achieves the minimax-optimal regret $\widetilde{\mathcal{O}}(d\sqrt{T})$ and supports infinite action spaces, outperforming UCB-based baselines in synthetic experiments. The paper also extends the framework to nonlinear reward settings, providing regret bounds that depend on a decoupling coefficient and priors, and offers a potential-based proof strategy that decouples the Bellman error from exploration. Empirically, FGTS.CDB demonstrates large-margin improvements and robustness to hyperparameters, highlighting the practical viability of Thompson sampling in contextual dueling bandits for scalable decision-making with preferences.

Abstract

Contextual dueling bandits, where a learner compares two options based on context and receives feedback indicating which was preferred, extends classic dueling bandits by incorporating contextual information for decision-making and preference learning. Several algorithms based on the upper confidence bound (UCB) have been proposed for linear contextual dueling bandits. However, no algorithm based on posterior sampling has been developed in this setting, despite the empirical success observed in traditional contextual bandits. In this paper, we propose a Thompson sampling algorithm, named FGTS.CDB, for linear contextual dueling bandits. At the core of our algorithm is a new Feel-Good exploration term specifically tailored for dueling bandits. This term leverages the independence of the two selected arms, thereby avoiding a cross term in the analysis. We show that our algorithm achieves nearly minimax-optimal regret, i.e., $\tilde{\mathcal{O}}(d\sqrt T)$, where $d$ is the model dimension and $T$ is the time horizon. Finally, we evaluate our algorithm on synthetic data and observe that FGTS.CDB outperforms existing algorithms by a large margin.

Feel-Good Thompson Sampling for Contextual Dueling Bandits

TL;DR

This work introduces FGTS.CDB, a Feel-Good Thompson sampling algorithm for linear contextual dueling bandits, addressing the absence of posterior-sampling methods in this setting. By integrating a Feel-Good exploration term and exploiting independence between the two arms, FGTS.CDB achieves the minimax-optimal regret and supports infinite action spaces, outperforming UCB-based baselines in synthetic experiments. The paper also extends the framework to nonlinear reward settings, providing regret bounds that depend on a decoupling coefficient and priors, and offers a potential-based proof strategy that decouples the Bellman error from exploration. Empirically, FGTS.CDB demonstrates large-margin improvements and robustness to hyperparameters, highlighting the practical viability of Thompson sampling in contextual dueling bandits for scalable decision-making with preferences.

Abstract

Contextual dueling bandits, where a learner compares two options based on context and receives feedback indicating which was preferred, extends classic dueling bandits by incorporating contextual information for decision-making and preference learning. Several algorithms based on the upper confidence bound (UCB) have been proposed for linear contextual dueling bandits. However, no algorithm based on posterior sampling has been developed in this setting, despite the empirical success observed in traditional contextual bandits. In this paper, we propose a Thompson sampling algorithm, named FGTS.CDB, for linear contextual dueling bandits. At the core of our algorithm is a new Feel-Good exploration term specifically tailored for dueling bandits. This term leverages the independence of the two selected arms, thereby avoiding a cross term in the analysis. We show that our algorithm achieves nearly minimax-optimal regret, i.e., , where is the model dimension and is the time horizon. Finally, we evaluate our algorithm on synthetic data and observe that FGTS.CDB outperforms existing algorithms by a large margin.
Paper Structure (24 sections, 15 theorems, 105 equations, 2 figures, 1 table, 1 algorithm)

This paper contains 24 sections, 15 theorems, 105 equations, 2 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Setting
  4. Algorithm Description
  5. Main Results
  6. Extension to Nonlinear Reward
  7. Overview of Proof
  8. Regret Decomposition
  9. Potential-Based Analysis
  10. Experiments
  11. Implementation Details of Different Algorithms
  12. Regret Comparison
  13. Ablation Study
  14. Conclusion
  15. Proof of Main Results
  16. ...and 9 more sections

Key Result

Theorem 5.2

Under Assumptions assumption:linear and assumption:lipschitz, assume that the hyperparameters are selected as $\eta=0.25$ and $\mu=1/(10e^B\sqrt T)$, then the expectation of the regret of Algorithm algorithm:FGTS.CDB satisfies

Figures (2)

  • Figure 1: Regret comparison with MaxInP, MaxPairUCB and CoLSTIM.
  • Figure 2: Performance of FGTS.CDB over different $\alpha$.

Theorems & Definitions (19)

  • Theorem 5.2
  • Theorem 5.3
  • Remark 5.4
  • Remark 5.5
  • Theorem 6.2
  • Remark 6.3
  • Remark 6.4
  • Lemma 7.1
  • Lemma 7.2
  • Theorem A.1: Restatement of Theorem \ref{['theorem:main']}
  • ...and 9 more