Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Algorithm Design and Stronger Guarantees for the Improving Multi-Armed Bandits Problem

Avrim Blum, Marten Garicano, Kavya Ravichandran, Dravyansh Sharma

TL;DR

This paper advances the improving multi-armed bandits framework by introducing parameterized algorithm families that exploit stronger concavity in arm reward curves, enabling data-dependent guarantees. It defines the Concavity Envelope Exponent and the PTRR$_\alpha$ family, proving $O(k^{\alpha/(\alpha+1)})$ competitive ratios for $\alpha>\beta_I$ and a matching lower bound, while also providing sample-complexity results via a data-driven Hyperparameter Transfer approach. It then proposes a Hybrid algorithm that achieves best-of-both-worlds BAI guarantees, certifying the true best arm on benign instances and delivering near-optimal worst-case performance otherwise, with polynomial sample complexity for tuning $\alpha$ and $B$ from offline data. Collectively, these contributions deliver practical, distribution-aware algorithms for IMAB with stronger data-dependent performance and clear pathways for tuning in real-world domains such as hyperparameter optimization and clinical trial design.

Abstract

The improving multi-armed bandits problem is a formal model for allocating effort under uncertainty, motivated by scenarios such as investing research effort into new technologies, performing clinical trials, and hyperparameter selection from learning curves. Each pull of an arm provides reward that increases monotonically with diminishing returns. A growing line of work has designed algorithms for improving bandits, albeit with somewhat pessimistic worst-case guarantees. Indeed, strong lower bounds of $Ω(k)$ and $Ω(\sqrt{k})$ multiplicative approximation factors are known for both deterministic and randomized algorithms (respectively) relative to the optimal arm, where $k$ is the number of bandit arms. In this work, we propose two new parameterized families of bandit algorithms and bound the sample complexity of learning the near-optimal algorithm from each family using offline data. The first family we define includes the optimal randomized algorithm from prior work. We show that an appropriately chosen algorithm from this family can achieve stronger guarantees, with optimal dependence on $k$, when the arm reward curves satisfy additional properties related to the strength of concavity. Our second family contains algorithms that both guarantee best-arm identification on well-behaved instances and revert to worst case guarantees on poorly-behaved instances. Taking a statistical learning perspective on the bandit rewards optimization problem, we achieve stronger data-dependent guarantees without the need for actually verifying whether the assumptions are satisfied.

Algorithm Design and Stronger Guarantees for the Improving Multi-Armed Bandits Problem

TL;DR

This paper advances the improving multi-armed bandits framework by introducing parameterized algorithm families that exploit stronger concavity in arm reward curves, enabling data-dependent guarantees. It defines the Concavity Envelope Exponent and the PTRR family, proving competitive ratios for and a matching lower bound, while also providing sample-complexity results via a data-driven Hyperparameter Transfer approach. It then proposes a Hybrid algorithm that achieves best-of-both-worlds BAI guarantees, certifying the true best arm on benign instances and delivering near-optimal worst-case performance otherwise, with polynomial sample complexity for tuning and from offline data. Collectively, these contributions deliver practical, distribution-aware algorithms for IMAB with stronger data-dependent performance and clear pathways for tuning in real-world domains such as hyperparameter optimization and clinical trial design.

Abstract

The improving multi-armed bandits problem is a formal model for allocating effort under uncertainty, motivated by scenarios such as investing research effort into new technologies, performing clinical trials, and hyperparameter selection from learning curves. Each pull of an arm provides reward that increases monotonically with diminishing returns. A growing line of work has designed algorithms for improving bandits, albeit with somewhat pessimistic worst-case guarantees. Indeed, strong lower bounds of and multiplicative approximation factors are known for both deterministic and randomized algorithms (respectively) relative to the optimal arm, where is the number of bandit arms. In this work, we propose two new parameterized families of bandit algorithms and bound the sample complexity of learning the near-optimal algorithm from each family using offline data. The first family we define includes the optimal randomized algorithm from prior work. We show that an appropriately chosen algorithm from this family can achieve stronger guarantees, with optimal dependence on , when the arm reward curves satisfy additional properties related to the strength of concavity. Our second family contains algorithms that both guarantee best-arm identification on well-behaved instances and revert to worst case guarantees on poorly-behaved instances. Taking a statistical learning perspective on the bandit rewards optimization problem, we achieve stronger data-dependent guarantees without the need for actually verifying whether the assumptions are satisfied.

Paper Structure

This paper contains 36 sections, 30 theorems, 85 equations, 4 algorithms.

Table of Contents

  1. Introduction
  2. Our Contributions
  3. Preliminaries
  4. Data-driven Algorithm Design Perspective and Problem Statement
  5. Exploiting Strength of Concavity
  6. Algorithm Family
  7. Sharper Competitive Ratio
  8. Sample Complexity Analysis
  9. Derandomized Dual Complexity and result of sharma2025offline.
  10. Bounding derandomized dual complexity in our setting.
  11. Sample complexity.
  12. Achieving Best-of-both-worlds BAI
  13. Hybrid Algorithm Family
  14. Best-of-Both-Worlds BAI guarantees
  15. Sample Complexity for tuning $\alpha,B$
  16. ...and 21 more sections

Key Result

Theorem 3.5

Given an IMAB instance $I$ with Concavity Envelope Exponent $\beta_I$. If $T \ge 2k$, then $PTRR_\alpha$ for $\alpha \in (\beta_I, 1)$ with $\tau =T-k$, $m = \frac{\tau}{T} f^*(T)$ achieves Equivalently, the competitive ratio is $O(k^{\alpha/(\alpha+1)})$. Setting $\alpha = 1$ recovers the $O(\sqrt{k})$ bound.

Theorems & Definitions (82)

  • Definition 2.1
  • Definition 2.2
  • Definition 2.3
  • Definition 2.4
  • Definition 2.5: Hyperparameter Transfer Setting
  • Definition 3.1
  • Definition 3.2: Per-arm $\beta$-Lower Envelope, LE($\beta$)
  • Definition 3.3: Concavity Envelope Exponent, $\beta_I$
  • Remark 3.4
  • Theorem 3.5
  • ...and 72 more