Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Best Arm Identification with Fixed Budget: A Large Deviation Perspective

Po-An Wang, Ruo-Chun Tzeng, Alexandre Proutiere

TL;DR

The paper tackles best-arm identification under a fixed budget in stochastic bandits by deriving a large-deviation framework that ties the LD behavior of adaptive arm-selection (ω(t)) to the LD behavior of empirical rewards (μ̂(t)). This connection yields new, tighter error-probability bounds and motivates truly adaptive algorithms. The authors introduce Continuous Rejects (CR) with two variants, CR-C and CR-A, showing they achieve higher exponential decay rates for the error probability than classic SR and SH methods, supported by both theory and extensive simulations. The work also refines SR’s performance via LD-based improvements and provides concrete rate expressions involving problem-dependent quantities. Overall, the approach advances understanding of instance-specific limits for adaptive best-arm identification and offers practical adaptive strategies with strong guarantees.

Abstract

We consider the problem of identifying the best arm in stochastic Multi-Armed Bandits (MABs) using a fixed sampling budget. Characterizing the minimal instance-specific error probability for this problem constitutes one of the important remaining open problems in MABs. When arms are selected using a static sampling strategy, the error probability decays exponentially with the number of samples at a rate that can be explicitly derived via Large Deviation techniques. Analyzing the performance of algorithms with adaptive sampling strategies is however much more challenging. In this paper, we establish a connection between the Large Deviation Principle (LDP) satisfied by the empirical proportions of arm draws and that satisfied by the empirical arm rewards. This connection holds for any adaptive algorithm, and is leveraged (i) to improve error probability upper bounds of some existing algorithms, such as the celebrated \sr (Successive Rejects) algorithm \citep{audibert2010best}, and (ii) to devise and analyze new algorithms. In particular, we present \sred (Continuous Rejects), a truly adaptive algorithm that can reject arms in {\it any} round based on the observed empirical gaps between the rewards of various arms. Applying our Large Deviation results, we prove that \sred enjoys better performance guarantees than existing algorithms, including \sr. Extensive numerical experiments confirm this observation.

Best Arm Identification with Fixed Budget: A Large Deviation Perspective

TL;DR

The paper tackles best-arm identification under a fixed budget in stochastic bandits by deriving a large-deviation framework that ties the LD behavior of adaptive arm-selection (ω(t)) to the LD behavior of empirical rewards (μ̂(t)). This connection yields new, tighter error-probability bounds and motivates truly adaptive algorithms. The authors introduce Continuous Rejects (CR) with two variants, CR-C and CR-A, showing they achieve higher exponential decay rates for the error probability than classic SR and SH methods, supported by both theory and extensive simulations. The work also refines SR’s performance via LD-based improvements and provides concrete rate expressions involving problem-dependent quantities. Overall, the approach advances understanding of instance-specific limits for adaptive best-arm identification and offers practical adaptive strategies with strong guarantees.

Abstract

We consider the problem of identifying the best arm in stochastic Multi-Armed Bandits (MABs) using a fixed sampling budget. Characterizing the minimal instance-specific error probability for this problem constitutes one of the important remaining open problems in MABs. When arms are selected using a static sampling strategy, the error probability decays exponentially with the number of samples at a rate that can be explicitly derived via Large Deviation techniques. Analyzing the performance of algorithms with adaptive sampling strategies is however much more challenging. In this paper, we establish a connection between the Large Deviation Principle (LDP) satisfied by the empirical proportions of arm draws and that satisfied by the empirical arm rewards. This connection holds for any adaptive algorithm, and is leveraged (i) to improve error probability upper bounds of some existing algorithms, such as the celebrated \sr (Successive Rejects) algorithm \citep{audibert2010best}, and (ii) to devise and analyze new algorithms. In particular, we present \sred (Continuous Rejects), a truly adaptive algorithm that can reject arms in {\it any} round based on the observed empirical gaps between the rewards of various arms. Applying our Large Deviation results, we prove that \sred enjoys better performance guarantees than existing algorithms, including \sr. Extensive numerical experiments confirm this observation.
Paper Structure (45 sections, 46 theorems, 231 equations, 12 figures, 7 tables, 2 algorithms)

This paper contains 45 sections, 46 theorems, 231 equations, 12 figures, 7 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Large Deviation Analysis of Adaptive Sampling Algorithms
  4. Large Deviation Principles
  5. Analysis of adaptive sampling algorithms
  6. Proof of Theorem \ref{['thm:Varadaham bandits']}
  7. Improved analysis of the Successive Rejects algorithm
  8. Continuous Rejects Algorithms
  9. The $\text{\tt CR-C}$ and $\text{\tt CR-A}$ algorithms
  10. Analysis of $\text{\tt CR-C}$ and $\text{\tt CR-A}$
  11. Numerical Experiments
  12. Conclusion
  13. Notation
  14. Technical lemmas towards the proof of Theorem \ref{['thm:Varadaham bandits']}
  15. Analysis of $\text{\tt CR}$
  16. ...and 30 more sections

Key Result

Theorem 1

Assume that under some adaptive sampling algorithm, $\{ \boldsymbol{\omega} (t)\}_{t\ge 1}$ satisfies the LDP upper bound (eq:rate UB) with rate function $I$. Then $\{\hat{\bm}(t)\}_{t\ge 1}$ satisfies the LDP upper bound (eq:rate UB) with rate function $\boldsymbol{\lambda}\mapsto \min_{\boldsymbol where $F_{ \mathcal{S}}(\boldsymbol{\omega}):=\inf_{\boldsymbol{\lambda} \in \textnormal{cl} (\math

Figures (12)

  • Figure 1: (One group of suboptimal arms) $\bm$ with $K=40$.
  • Figure 2: (One group of suboptimal arms) error probabilities averaged over $40,000$ independent runs.
  • Figure 3: (Two groups of suboptimal arms) $\bm$ with $K=40$.
  • Figure 4: (Two groups of suboptimal arms) error probabilities averaged over $40,000$ independent runs.
  • Figure 5: (Linear arm-to-reward function) $\bm$ with $K=40$.
  • ...and 7 more figures

Theorems & Definitions (87)

  • Definition 1
  • Theorem 1
  • proof
  • Lemma 1
  • proof
  • Theorem 2
  • Theorem 3
  • Theorem 4
  • Lemma 2
  • proof
  • ...and 77 more