Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fast and Regret Optimal Best Arm Identification: Fundamental Limits and Low-Complexity Algorithms

Qining Zhang, Lei Ying

TL;DR

This work studies Regret Optimal Best Arm Identification (ROBAI), a dual-objective stochastic bandit problem balancing fast commitment to the optimal arm with low cumulative regret. It introduces EOCP, a low-complexity Explore-Optimistically-then-Commit algorithm, and its adaptive variant EOCP-UG, achieving regret-optimality in Gaussian bandits and provable sample-complexity bounds, including $O( ext{log}T)$ commitment for pre-determined stopping and $O( ext{log}^2T)$ for adaptive stopping, both with $O(T^{-1})$ confidence. The paper further derives fundamental limits on the sample complexity until commitment and presents KL-EOCP to extend regret-optimality to general exponential-family bandits, matching asymptotic lower bounds under suitable KL-divergence bounds. Numerical experiments corroborate the theory and highlight an over-exploration phenomenon in classic UCB methods, illustrating the practical benefit of the proposed pessimistic-LCB commitment strategy. Overall, the results offer a principled, sample-efficient framework for achieving fast, reliable commitment to the best arm while maintaining strong regret guarantees in both Gaussian and general bandit settings.

Abstract

This paper considers a stochastic Multi-Armed Bandit (MAB) problem with dual objectives: (i) quick identification and commitment to the optimal arm, and (ii) reward maximization throughout a sequence of $T$ consecutive rounds. Though each objective has been individually well-studied, i.e., best arm identification for (i) and regret minimization for (ii), the simultaneous realization of both objectives remains an open problem, despite its practical importance. This paper introduces \emph{Regret Optimal Best Arm Identification} (ROBAI) which aims to achieve these dual objectives. To solve ROBAI with both pre-determined stopping time and adaptive stopping time requirements, we present an algorithm called EOCP and its variants respectively, which not only achieve asymptotic optimal regret in both Gaussian and general bandits, but also commit to the optimal arm in $\mathcal{O}(\log T)$ rounds with pre-determined stopping time and $\mathcal{O}(\log^2 T)$ rounds with adaptive stopping time. We further characterize lower bounds on the commitment time (equivalent to the sample complexity) of ROBAI, showing that EOCP and its variants are sample optimal with pre-determined stopping time, and almost sample optimal with adaptive stopping time. Numerical results confirm our theoretical analysis and reveal an interesting "over-exploration" phenomenon carried by classic UCB algorithms, such that EOCP has smaller regret even though it stops exploration much earlier than UCB, i.e., $\mathcal{O}(\log T)$ versus $\mathcal{O}(T)$, which suggests over-exploration is unnecessary and potentially harmful to system performance.

Fast and Regret Optimal Best Arm Identification: Fundamental Limits and Low-Complexity Algorithms

TL;DR

This work studies Regret Optimal Best Arm Identification (ROBAI), a dual-objective stochastic bandit problem balancing fast commitment to the optimal arm with low cumulative regret. It introduces EOCP, a low-complexity Explore-Optimistically-then-Commit algorithm, and its adaptive variant EOCP-UG, achieving regret-optimality in Gaussian bandits and provable sample-complexity bounds, including commitment for pre-determined stopping and for adaptive stopping, both with confidence. The paper further derives fundamental limits on the sample complexity until commitment and presents KL-EOCP to extend regret-optimality to general exponential-family bandits, matching asymptotic lower bounds under suitable KL-divergence bounds. Numerical experiments corroborate the theory and highlight an over-exploration phenomenon in classic UCB methods, illustrating the practical benefit of the proposed pessimistic-LCB commitment strategy. Overall, the results offer a principled, sample-efficient framework for achieving fast, reliable commitment to the best arm while maintaining strong regret guarantees in both Gaussian and general bandit settings.

Abstract

This paper considers a stochastic Multi-Armed Bandit (MAB) problem with dual objectives: (i) quick identification and commitment to the optimal arm, and (ii) reward maximization throughout a sequence of consecutive rounds. Though each objective has been individually well-studied, i.e., best arm identification for (i) and regret minimization for (ii), the simultaneous realization of both objectives remains an open problem, despite its practical importance. This paper introduces \emph{Regret Optimal Best Arm Identification} (ROBAI) which aims to achieve these dual objectives. To solve ROBAI with both pre-determined stopping time and adaptive stopping time requirements, we present an algorithm called EOCP and its variants respectively, which not only achieve asymptotic optimal regret in both Gaussian and general bandits, but also commit to the optimal arm in rounds with pre-determined stopping time and rounds with adaptive stopping time. We further characterize lower bounds on the commitment time (equivalent to the sample complexity) of ROBAI, showing that EOCP and its variants are sample optimal with pre-determined stopping time, and almost sample optimal with adaptive stopping time. Numerical results confirm our theoretical analysis and reveal an interesting "over-exploration" phenomenon carried by classic UCB algorithms, such that EOCP has smaller regret even though it stops exploration much earlier than UCB, i.e., versus , which suggests over-exploration is unnecessary and potentially harmful to system performance.
Paper Structure (33 sections, 17 theorems, 207 equations, 2 figures, 1 table, 3 algorithms)

This paper contains 33 sections, 17 theorems, 207 equations, 2 figures, 1 table, 3 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Low-Complexity Algorithms
  4. The Pre-determined Stopping Time Setting
  5. The Adaptive Stopping Time Setting
  6. Main Results
  7. Regret Optimality for Gaussian Bandits with Pre-Determined Stopping Time
  8. Regret Optimality for Gaussian Bandits with Adaptive Stopping Time
  9. Fundamental Limits of Sample Complexity until Commitment
  10. Regret Optimality for General Bandits
  11. Numerical Experiments
  12. Conclusion
  13. Acknowledgement
  14. Related Works
  15. Regret Minimization
  16. ...and 18 more sections

Key Result

Theorem 1

Let $l = \log(T) + \mathcal{O}(\sqrt{\log T})$ and when $T$ is large enough, the expected regret of the EOCP algorithm in Algorithm. alg:fixed with pre-determined stopping time can be upper-bounded by:

Figures (2)

  • Figure 1: Comparison of regret performance of EOCP with variants and existing algorithms in the literature. The gap $\Delta$ between the two arms is $0.5$ and results are averaged over $10^5$ iterations.
  • Figure 2: Regret performance in bandit models with $4$ arms.

Theorems & Definitions (17)

  • Theorem 1
  • Corollary 1
  • Theorem 2
  • Corollary 2
  • Theorem 3: Information-Theoretic Limits of $\mathsf{SCC}$
  • Theorem 4
  • Lemma 1: Theorem 9.2 in lattimore2020banditbook
  • Lemma 2: Lemma C.3 in jin2021doubleETC
  • Lemma 3
  • Theorem 5
  • ...and 7 more