Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Lipschitz Dueling Bandits over Continuous Action Spaces

Mudit Sharma, Shweta Jain, Vaneet Aggarwal, Ganesh Ghalme

Abstract

We study for the first time, stochastic dueling bandits over continuous action spaces with Lipschitz structure, where feedback is purely comparative. While dueling bandits and Lipschitz bandits have been studied separately, their combination has remained unexplored. We propose the first algorithm for Lipschitz dueling bandits, using round-based exploration and recursive region elimination guided by an adaptive reference arm. We develop new analytical tools for relative feedback and prove a regret bound of $\tilde O\left(T^{\frac{d_z+1}{d_z+2}}\right)$, where $d_z$ is the zooming dimension of the near-optimal region. Further, our algorithm takes only logarithmic space in terms of the total time horizon, best achievable by any bandit algorithm over a continuous action space.

Lipschitz Dueling Bandits over Continuous Action Spaces

Abstract

We study for the first time, stochastic dueling bandits over continuous action spaces with Lipschitz structure, where feedback is purely comparative. While dueling bandits and Lipschitz bandits have been studied separately, their combination has remained unexplored. We propose the first algorithm for Lipschitz dueling bandits, using round-based exploration and recursive region elimination guided by an adaptive reference arm. We develop new analytical tools for relative feedback and prove a regret bound of , where is the zooming dimension of the near-optimal region. Further, our algorithm takes only logarithmic space in terms of the total time horizon, best achievable by any bandit algorithm over a continuous action space.

Paper Structure

This paper contains 15 sections, 9 theorems, 70 equations, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Dueling Bandits and Preference-Based Bandits.
  4. Lipschitz and Continuum-Armed Bandits.
  5. Problem Setup and Regret Definition
  6. Regret in Lipschitz Dueling Bandits
  7. Algorithm
  8. Regret Analysis for Log-DuelLi
  9. Memory usage of Algorithm \ref{['alg:logli']}.
  10. Memory usage of Algorithm \ref{['alg:roundfunc']}.
  11. Encoding cost.
  12. Lower Bounds
  13. Conclusion
  14. Lipschitz continuity of transfer functions
  15. Stochastic triangle inequality for standard transfer functions

Key Result

Lemma 5.1

Let $(\mathcal{X},\|\cdot\|)$ be a metric space and let $f:\mathcal{X}\to\mathbb{R}$ be $1$-Lipschitz. Assume pairwise preferences are generated according to where $\rho(z)$ is a globally Lipschitz transfer function. Fix a round $m$, depth $h$, and a cube $C\in\mathcal{A}_h^m$ of diameter at most $r_h$. Let $X^{m-1}$ be the reference arm and let $\{x_i\}_{i=1}^{n_h}$ be i.i.d. samples drawn unifo

Theorems & Definitions (18)

  • Lemma 5.1: Uniform concentration for cube-level preference estimates
  • proof
  • Lemma 5.2: Optimal arm is never eliminated
  • proof
  • Lemma 5.3: Shrinking-region property for dueling bandits
  • proof
  • Theorem 5.4: Regret bound for dueling Lipschitz bandits
  • proof
  • Lemma 5.5: Space complexity
  • proof
  • ...and 8 more