Optimism in the Face of Ambiguity Principle for Multi-Armed Bandits
Mengmeng Li, Daniel Kuhn, Bahar Taşkesen
TL;DR
The paper tackles the challenge of achieving best-of-both-worlds regret in multi-armed bandits while preserving computational efficiency. It introduces Distributionally Optimistic Perturbations (DOPA), a GBPA variant that uses marginal ambiguity sets to induce optimistic perturbations, linking FTPL and FTRL via an additively separable regularizer $\psi$ and the gradient relationship $\nabla_{\boldsymbol u} \Phi^R(\boldsymbol u; \psi)=\nabla_{\boldsymbol u} \Phi(\boldsymbol u; \mathcal{B})$. By proving equivalence between FTRL with separable regularizers and FTPL under appropriate ambiguity sets, and by deriving regret bounds under Fréchet marginals (e.g., shifted Pareto) that achieve $\mathcal{R}(T)=\mathcal{O}(\sqrt{KT})$ in adversarial settings and $\mathcal{O}(\log T)$ in stochastic settings, the approach unifies the two paradigms with a computationally efficient perturbation mechanism. A fast, bisection-based method computes arm-sampling probabilities with $\mathcal{O}(K)$ per-iteration complexity, yielding up to $10^4$× speedups over classical FTRL while retaining BOBW guarantees, and enabling broader applicability to hybrid regularizers and beyond-bandit problems. The work thus provides a practical, theory-backed bridge between regularization and perturbation in online learning.
Abstract
Follow-The-Regularized-Leader (FTRL) algorithms often enjoy optimal regret for adversarial as well as stochastic bandit problems and allow for a streamlined analysis. Nonetheless, FTRL algorithms require the solution of an optimization problem in every iteration and are thus computationally challenging. In contrast, Follow-The-Perturbed-Leader (FTPL) algorithms achieve computational efficiency by perturbing the estimates of the rewards of the arms, but their regret analysis is cumbersome. We propose a new FTPL algorithm that generates optimal policies for both adversarial and stochastic multi-armed bandits. Like FTRL, our algorithm admits a unified regret analysis, and similar to FTPL, it offers low computational costs. Unlike existing FTPL algorithms that rely on independent additive disturbances governed by a \textit{known} distribution, we allow for disturbances governed by an \textit{ambiguous} distribution that is only known to belong to a given set and propose a principle of optimism in the face of ambiguity. Consequently, our framework generalizes existing FTPL algorithms. It also encapsulates a broad range of FTRL methods as special cases, including several optimal ones, which appears to be impossible with current FTPL methods. Finally, we use techniques from discrete choice theory to devise an efficient bisection algorithm for computing the optimistic arm sampling probabilities. This algorithm is up to $10^4$ times faster than standard FTRL algorithms that solve an optimization problem in every iteration. Our results not only settle existing conjectures but also provide new insights into the impact of perturbations by mapping FTRL to FTPL.