Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Quantile Multi-Armed Bandits with 1-bit Feedback

Ivan Lau, Jonathan Scarlett

TL;DR

This work studies best-arm identification for quantile rewards under a strict 1-bit communication constraint. The authors develop a successive-elimination algorithm that uses a quantile-estimation subroutine (QuantEst) based on noisy binary search to produce confidence intervals for each arm's $q$-quantile, all while exchanging only 1-bit information per pull. They establish an instance-dependent upper bound on the sample complexity that scales with $ ilde{O}ig( rac{1}{ ext{gap}^2}ig)$ factors and $ ilde{O}ig( ext{log}( rac{oldsymbol{ extlambda}}{oldsymbol{ extepsilon}})ig)$ dependence, and provide matching lower bounds showing that 1-bit feedback has limited impact on scaling, especially under threshold-query constraints which incur at least $oldsymbol{ extOmega}( ext{log}(oldsymbol{ extlambda}/oldsymbol{ extepsilon}))$ samples. The paper also characterizes solvable versus unsolvable instances, proving that positive gaps ensure solvability while zero-gap instances are generally unsolvable unless the gap definition is adjusted. Overall, the results illuminate how quantile-based objectives can be effectively tackled under severe communication constraints, with tight bounds across a broad regime of problem instances.

Abstract

In this paper, we study a variant of best-arm identification involving elements of risk sensitivity and communication constraints. Specifically, the goal of the learner is to identify the arm with the highest quantile reward, while the communication from an agent (who observes rewards) and the learner (who chooses actions) is restricted to only one bit of feedback per arm pull. We propose an algorithm that utilizes noisy binary search as a subroutine, allowing the learner to estimate quantile rewards through 1-bit feedback. We derive an instance-dependent upper bound on the sample complexity of our algorithm and provide an algorithm-independent lower bound for specific instances, with the two matching to within logarithmic factors under mild conditions, or even to within constant factors in certain low error probability scaling regimes. The lower bound is applicable even in the absence of communication constraints, and thus we conclude that restricting to 1-bit feedback has a minimal impact on the scaling of the sample complexity.

Quantile Multi-Armed Bandits with 1-bit Feedback

TL;DR

This work studies best-arm identification for quantile rewards under a strict 1-bit communication constraint. The authors develop a successive-elimination algorithm that uses a quantile-estimation subroutine (QuantEst) based on noisy binary search to produce confidence intervals for each arm's -quantile, all while exchanging only 1-bit information per pull. They establish an instance-dependent upper bound on the sample complexity that scales with factors and dependence, and provide matching lower bounds showing that 1-bit feedback has limited impact on scaling, especially under threshold-query constraints which incur at least samples. The paper also characterizes solvable versus unsolvable instances, proving that positive gaps ensure solvability while zero-gap instances are generally unsolvable unless the gap definition is adjusted. Overall, the results illuminate how quantile-based objectives can be effectively tackled under severe communication constraints, with tight bounds across a broad regime of problem instances.

Abstract

In this paper, we study a variant of best-arm identification involving elements of risk sensitivity and communication constraints. Specifically, the goal of the learner is to identify the arm with the highest quantile reward, while the communication from an agent (who observes rewards) and the learner (who chooses actions) is restricted to only one bit of feedback per arm pull. We propose an algorithm that utilizes noisy binary search as a subroutine, allowing the learner to estimate quantile rewards through 1-bit feedback. We derive an instance-dependent upper bound on the sample complexity of our algorithm and provide an algorithm-independent lower bound for specific instances, with the two matching to within logarithmic factors under mild conditions, or even to within constant factors in certain low error probability scaling regimes. The lower bound is applicable even in the absence of communication constraints, and thus we conclude that restricting to 1-bit feedback has a minimal impact on the scaling of the sample complexity.

Paper Structure

This paper contains 30 sections, 22 theorems, 99 equations, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Setup and Contributions
  4. Problem Setup
  5. Summary of Contributions.
  6. Algorithm and Upper Bound
  7. Description of the Algorithm
  8. Anytime Quantile Bounds
  9. Correctness
  10. Upper Bound
  11. Lower Bounds
  12. Lower Bound for the Unquantized Variant
  13. $\Omega(\log(\lambda/\epsilon))$ Dependence Under Threshold Query Model
  14. Solvable Instances
  15. Quantile Estimation Subroutine
  16. ...and 15 more sections

Key Result

Lemma 2

Fix an instance $\nu \in \mathcal{E}$, and suppose Algorithm alg: main is run with input $(\mathcal{A}, \lambda, \epsilon, q, \delta)$ and parameter $c \ge 1$. Let $\Delta^{(t)}$, $\mathcal{A}_t$, $l_{t, k}$, $u_{t, k}$ be as defined in Algorithm alg: main for each round index $t \ge 1$ and each arm and Then the Event $E$ defined by occurs with probability at least $1 - \delta$. Furthermore, for

Theorems & Definitions (41)

  • Remark 1
  • Lemma 2: Good event
  • Remark 3
  • Lemma 4: Anytime quantile bounds
  • Remark 5
  • Definition 6: $(\epsilon, \delta)$-reliable.
  • Remark 7
  • Theorem 8: Reliability of Algorithm \ref{['alg: main']}
  • Corollary 9
  • Definition 10: Arm gaps
  • ...and 31 more