Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adaptive Algorithms for Infinitely Many-Armed Bandits: A Unified Framework

Emmanuel Pilliat

TL;DR

The paper studies infinitely many-armed bandits with a limited budget, aiming to maximize the expected simple reward of the recommended arm under possibly unbounded arm-mean distributions, and it does so with anytime guarantees. It introduces OSE, a distribution-free algorithm, and PROSE, a practical enhancement, both analyzed through a rank-based framework centered on the rank-corrected inverse squared gap function $G$ and the threshold function $\eta^*_t(\psi)$. The main result shows that OSE achieves high-probability top-rank recommendations, recovering known rates for Beta-type and Pareto-type distributions and revealing new transition regimes dependent on tail parameter $\alpha$ and noise level $\zeta$. The work also delivers an efficient implementation with amortized $O(\log^2 t)$ updates and demonstrates strong empirical performance, bridging theory and practice for massive arm reservoirs.

Abstract

We consider a bandit problem where the buget is smaller than the number of arms, which may be infinite. In this regime, the usual objective in the literature is to minimize simple regret. To analyze broad classes of distributions with potentially unbounded support, where simple regret may not be well-defined, we take a slightly different approach and seek to maximize the expected simple reward of the recommended arm, providing anytime guarantees. To that end, we introduce a distribution-free algorithm, OSE, that adapts to the distribution of arm means and achieves near-optimal rates for several distribution classes. We characterize the sample complexity through the rank-corrected inverse squared gap function. In particular, we recover known upper bounds and transition regimes for $α$ less or greater than $1/2$ when the quantile function is $λ_η= 1-η^α$. We additionally identify new transition regimes depending on the noise level relative to $α$, which we conjecture to be nearly optimal. Additionally, we introduce an enhanced practical version, PROSE, that achieves state-of-the-art empirical performance for the main distribution classes considered in the literature.

Adaptive Algorithms for Infinitely Many-Armed Bandits: A Unified Framework

TL;DR

The paper studies infinitely many-armed bandits with a limited budget, aiming to maximize the expected simple reward of the recommended arm under possibly unbounded arm-mean distributions, and it does so with anytime guarantees. It introduces OSE, a distribution-free algorithm, and PROSE, a practical enhancement, both analyzed through a rank-based framework centered on the rank-corrected inverse squared gap function and the threshold function . The main result shows that OSE achieves high-probability top-rank recommendations, recovering known rates for Beta-type and Pareto-type distributions and revealing new transition regimes dependent on tail parameter and noise level . The work also delivers an efficient implementation with amortized updates and demonstrates strong empirical performance, bridging theory and practice for massive arm reservoirs.

Abstract

We consider a bandit problem where the buget is smaller than the number of arms, which may be infinite. In this regime, the usual objective in the literature is to minimize simple regret. To analyze broad classes of distributions with potentially unbounded support, where simple regret may not be well-defined, we take a slightly different approach and seek to maximize the expected simple reward of the recommended arm, providing anytime guarantees. To that end, we introduce a distribution-free algorithm, OSE, that adapts to the distribution of arm means and achieves near-optimal rates for several distribution classes. We characterize the sample complexity through the rank-corrected inverse squared gap function. In particular, we recover known upper bounds and transition regimes for less or greater than when the quantile function is . We additionally identify new transition regimes depending on the noise level relative to , which we conjecture to be nearly optimal. Additionally, we introduce an enhanced practical version, PROSE, that achieves state-of-the-art empirical performance for the main distribution classes considered in the literature.

Paper Structure

This paper contains 18 sections, 6 theorems, 49 equations, 5 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Contributions
  4. Technical Overview
  5. Main Result
  6. Algorithms
  7. Implications of the Main Theorem
  8. Uniform $\epsilon$-error probability bound
  9. Applications to Various Distribution Types
  10. Numerical Study
  11. Proof
  12. High Proability Events
  13. Analysis of the Procedure, Proof of \ref{['th:main_thm_new']}
  14. Step 1: Existence of a top-$\rho$ arm with small exploration scope.
  15. Step 2: Lower and upper bounds on the $\mathrm{UCB}$'s for all arms.
  16. ...and 3 more sections

Key Result

Theorem 2.1

Fix any $\delta \in (0,1)$ and $t \geq 1$. Assume that $\psi :=\psi_{t,\delta} \geq 2^{30} \log^3(5t/\delta)$, and set the tuning parameter $\beta_t = 6\log(5t/\delta)$. With probability $1-\delta$, for all $t \geq 1$, the recommendation $\hat{r}_t$ of algo:ose ($\mathbf{OSE}$) defined in sec:algos

Figures (5)

  • Figure 1: Probability density functions of $\mathrm{Beta}(\gamma, 1)$ distributions for various values of $\gamma$. Higher values of $\gamma$ concentrate the distribution toward 1, corresponding to more aggressive exploration strategies.
  • Figure 2: Probability density functions for Beta$(1, 1/\alpha)$ (left) and Pareto$(1/\alpha, 1)$ (right) distributions for various values of $\alpha$. The Beta distribution is supported on $[0,1]$ and becomes more concentrated near 1 as $\alpha$ increases. The Pareto distribution is supported on $[1, \infty)$ and has heavier tail as $\alpha$ decreases.
  • Figure 3: Expected simple regrets for the four algorithms as a function of $t$ on a log scale. Each plot shows results for quantile function $\lambda_{\eta} = 1-\eta^{\alpha}$ with $\alpha \in \{0.25, 0.5, 1, 2\}$, using $K = 5000$ arms and time horizon $T_{\mathrm{max}}=50000$. Trajectories display median simple regret over $N=2000$ trials. Shaded regions (blue for $\mathbf{PROSE}$, orange for $\mathbf{BSH}$) show the interquartile range.
  • Figure 4: Expected simple regrets for $\mathbf{BSH}$, $\mathbf{PROSE}$ (left), and $\mathbf{OSE}$ (right) with tuning parameter $\beta \in \{1, 5, 10, 50\}$, for $\mathrm{Beta}(1, 1/\alpha)$ instances with $\alpha \in \{0.5, 1\}$. Parameters: $K=5000$ arms, time horizon $T_{\mathrm{max}}= 50000$, and $N=2000$ trials per trajectory.
  • Figure 5: Cumulative regrets for $\mathbf{BSH}$, $\mathbf{PROSE}$, $\mathbf{OSE}$, and UCB on $\mathrm{Beta}(1, 1/\alpha)$ instances with $\alpha \in \{0.5, 1\}$. $\mathbf{PROSE}$ follows the same long-term sublinear trend as UCB, while $\mathbf{OSE}$ and $\mathbf{BSH}$ exhibit quasi-linear growth. Parameters: $K=5000$ arms, $T_{\mathrm{max}}= 50000$, $N=2000$ trials.

Theorems & Definitions (12)

  • Definition 1: $\delta$-achievable rank sequence
  • Definition 2: $\psi$-significant rank at timestep $t$
  • Theorem 2.1
  • Corollary 2.2
  • Corollary 4.1
  • proof : Proof of \ref{['cor:epsilon_gaps']}
  • Lemma A.1
  • Lemma A.2
  • Lemma A.3
  • proof : Proof of \ref{['lem:concentration_noise']}
  • ...and 2 more