Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Replicability is Asymptotically Free in Multi-armed Bandits

Junpei Komiyama, Shinji Ito, Yuichi Yoshida, Souta Koshino

TL;DR

This paper addresses replicability in stochastic multi-armed bandits by introducing randomized, phased decision mechanisms that bound the probability of nonreplication $\rho$. It presents two main replicable algorithms, REC and RSE, achieving regret close to nonreplicable optimal rates and proving a lower bound for the two-armed case. The authors extend to linear bandits with RLSE, leveraging G-optimal design to maintain replicability while attaining near-optimal, dimension-dependent regret. A general bound framework decomposes replicability costs and shows these costs vanish asymptotically as $T$ grows, enhancing the practical viability of replicable sequential experiments. Simulations corroborate improvements over prior replicable approaches and illustrate the scalability to higher dimensions and richer reward structures.

Abstract

We consider a replicable stochastic multi-armed bandit algorithm that ensures, with high probability, that the algorithm's sequence of actions is not affected by the randomness inherent in the dataset. Replicability allows third parties to reproduce published findings and assists the original researcher in applying standard statistical tests. We observe that existing algorithms require $O(K^2/ρ^2)$ times more regret than nonreplicable algorithms, where $K$ is the number of arms and $ρ$ is the level of nonreplication. However, we demonstrate that this additional cost is unnecessary when the time horizon $T$ is sufficiently large for a given $K, ρ$, provided that the magnitude of the confidence bounds is chosen carefully. Therefore, for a large $T$, our algorithm only suffers $K^2/ρ^2$ times smaller amount of exploration than existing algorithms. To ensure the replicability of the proposed algorithms, we incorporate randomness into their decision-making processes. We propose a principled approach to limiting the probability of nonreplication. This approach elucidates the steps that existing research has implicitly followed. Furthermore, we derive the first lower bound for the two-armed replicable bandit problem, which implies the optimality of the proposed algorithms up to a $\log\log T$ factor for the two-armed case.

Replicability is Asymptotically Free in Multi-armed Bandits

TL;DR

This paper addresses replicability in stochastic multi-armed bandits by introducing randomized, phased decision mechanisms that bound the probability of nonreplication . It presents two main replicable algorithms, REC and RSE, achieving regret close to nonreplicable optimal rates and proving a lower bound for the two-armed case. The authors extend to linear bandits with RLSE, leveraging G-optimal design to maintain replicability while attaining near-optimal, dimension-dependent regret. A general bound framework decomposes replicability costs and shows these costs vanish asymptotically as grows, enhancing the practical viability of replicable sequential experiments. Simulations corroborate improvements over prior replicable approaches and illustrate the scalability to higher dimensions and richer reward structures.

Abstract

We consider a replicable stochastic multi-armed bandit algorithm that ensures, with high probability, that the algorithm's sequence of actions is not affected by the randomness inherent in the dataset. Replicability allows third parties to reproduce published findings and assists the original researcher in applying standard statistical tests. We observe that existing algorithms require times more regret than nonreplicable algorithms, where is the number of arms and is the level of nonreplication. However, we demonstrate that this additional cost is unnecessary when the time horizon is sufficiently large for a given , provided that the magnitude of the confidence bounds is chosen carefully. Therefore, for a large , our algorithm only suffers times smaller amount of exploration than existing algorithms. To ensure the replicability of the proposed algorithms, we incorporate randomness into their decision-making processes. We propose a principled approach to limiting the probability of nonreplication. This approach elucidates the steps that existing research has implicitly followed. Furthermore, we derive the first lower bound for the two-armed replicable bandit problem, which implies the optimality of the proposed algorithms up to a factor for the two-armed case.
Paper Structure (31 sections, 18 theorems, 82 equations, 1 figure, 3 tables, 3 algorithms)

This paper contains 31 sections, 18 theorems, 82 equations, 1 figure, 3 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Replicability
  3. Main results
  4. Problem Setup
  5. General Bound of the Probability of Nonreplication $\rho$
  6. An $O(K)$-regret Algorithm for $K$-armed Bandit Problem
  7. A Generalized Algorithm for $K$-armed Bandit Problem
  8. Regret Lower Bound for Replicable Algorithms
  9. An Algorithm for Linear Bandit Problem
  10. Simulation
  11. Related Work
  12. Stability in Sequential Learning
  13. Batched Bandit Problem
  14. Conclusion
  15. Notation
  16. ...and 16 more sections

Key Result

Lemma 1

Let $X_1,X_2,\ldots,X_N$ be $N$ independent (zero-mean) $\sigma$-subgaussian random variables, and $\hat{\mu}_N = (1/N)\sum_i X_i$ be the empirical mean. Then, for any $\varepsilon>0$ we have

Figures (1)

  • Figure 1: Regret of algorithms. The horizontal axis indicates the number of rounds $t$ from $1$ to $T$, whereas the vertical axis indicates $\mathrm{Reg}(t)$. Results of REC and RSE in Model 1 are very similar.

Theorems & Definitions (44)

  • Example 1
  • Lemma 1: Concentration inequality
  • Definition 2
  • Definition 3
  • Definition 4
  • Definition 5
  • Definition 6
  • Example 2
  • Definition 7
  • Definition 8
  • ...and 34 more