Best Arm Identification with Minimal Regret

TL;DR

Focusing on single-parameter exponential families of distributions, this work designs and analyzes the Double KL-UCB algorithm, which achieves asymptotic optimality as the confidence level tends to zero, and elucidate a fresh perspective on the inherent connections between regret minimization and BAI.

Abstract

Motivated by real-world applications that necessitate responsible experimentation, we introduce the problem of best arm identification (BAI) with minimal regret. This innovative variant of the multi-armed bandit problem elegantly amalgamates two of its most ubiquitous objectives: regret minimization and BAI. More precisely, the agent's goal is to identify the best arm with a prescribed confidence level $δ$, while minimizing the cumulative regret up to the stopping time. Focusing on single-parameter exponential families of distributions, we leverage information-theoretic techniques to establish an instance-dependent lower bound on the expected cumulative regret. Moreover, we present an intriguing impossibility result that underscores the tension between cumulative regret and sample complexity in fixed-confidence BAI. Complementarily, we design and analyze the Double KL-UCB algorithm, which achieves asymptotic optimality as the confidence level tends to zero. Notably, this algorithm employs two distinct confidence bounds to guide arm selection in a randomized manner. Our findings elucidate a fresh perspective on the inherent connections between regret minimization and BAI.

Theorems & Definitions (20)

• Definition 1
• Remark 2
• Theorem 3: Information-theoretic lower bound
• Definition 4: Asymptotic optimality
• Theorem 5: Impossibility result
• Remark 6
• Theorem 7
• Example 1
• Lemma 8: menard2017minimax
• Lemma 9: Maximal Inequality menard2017minimax