On Instability of Minimax Optimal Optimism-Based Bandit Algorithms

TL;DR

The paper analyzes the statistical stability of optimism-based multi-armed bandit algorithms and shows a fundamental tension between minimax-optimal regret and stability. By establishing general sufficient conditions (Assumptions A, H-G, and a sandwich bound on U_a(t)) under which optimism-based policies fail Lai–Wei stability, the authors prove that the entire MOSS-family and KL-based minimax-optimal variants are unstable for common reward models, with simulations confirming non-normal limits for sample means. This work implies that simultaneous achievement of minimax optimality and asymptotic normality under adaptive data collection may be impossible within this class, raising open questions about designing stable yet optimally performing bandits. The findings have practical implications for inference in adaptive experiments and motivate developing new strategies that preserve stability without sacrificing regret performance.

Abstract

Statistical inference from data generated by multi-armed bandit (MAB) algorithms is challenging due to their adaptive, non-i.i.d. nature. A classical manifestation is that sample averages of arm rewards under bandit sampling may fail to satisfy a central limit theorem. Lai and Wei's stability condition provides a sufficient, and essentially necessary criterion, for asymptotic normality in bandit problems. While the celebrated Upper Confidence Bound (UCB) algorithm satisfies this stability condition, it is not minimax optimal, raising the question of whether minimax optimality and statistical stability can be achieved simultaneously. In this paper, we analyze the stability properties of a broad class of bandit algorithms that are based on the optimism principle. We establish general structural conditions under which such algorithms violate the Lai-Wei stability criterion. As a consequence, we show that widely used minimax-optimal UCB-style algorithms, including MOSS, Anytime-MOSS, Vanilla-MOSS, ADA-UCB, OC-UCB, KL-MOSS, KL-UCB++, KL-UCB-SWITCH, and Anytime KL-UCB-SWITCH, are unstable. We further complement our theoretical results with numerical simulations demonstrating that, in all these cases, the sample means fail to exhibit asymptotic normality. Overall, our findings suggest a fundamental tension between stability and minimax optimal regret, raising the question of whether it is possible to design bandit algorithms that achieve both. Understanding whether such simultaneously stable and minimax optimal strategies exist remains an important open direction.

Figures (10)

• Figure 1: Left: Empirical results for UCB-$1$ based on $5000$ independent sample runs with $T = 10000$ pulls. Right: Coverage of Wald confidence interval of MOSS with $T = 10000$.
• Figure 2: Left: Empirical results for MOSS based on $5000$ independent sample runs with $T = 10000$ pulls. Right: Coverage of Wald confidence interval of MOSS with $T = 10000$.
• Figure 3: Left: Empirical results for Vanilla MOSS based on $5000$ independent sample runs with $T = 10000$ pulls. Right: Coverage of Wald confidence interval of Vanilla MOSS with $T = 10000$.
• Figure 4: Left: Empirical results for Anytime MOSS based on $5000$ independent sample runs with $T = 10000$ pulls. Right: Coverage of Wald confidence interval of Anytime MOSS with $T = 10000$.
• Figure 5: Left: Empirical results for KL-MOSS based on $5000$ independent sample runs with $T = 10000$ pulls. Right: Coverage of Wald confidence interval of KL-MOSS with $T = 10000$.

Theorems & Definitions (24)

• Definition 1
• Definition 2: Kullback--Leibler Divergence
• Lemma 1
• Theorem 1
• Corollary 1
• Corollary 2
• Lemma 2
• Lemma 3
• Lemma 4
• Lemma 5