Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploration in the Limit

Brian M. Cho, Nathan Kallus

Abstract

In fixed-confidence best arm identification (BAI), the objective is to quickly identify the optimal option while controlling the probability of error below a desired threshold. Despite the plethora of BAI algorithms, existing methods typically fall short in practical settings, as stringent exact error control requires using loose tail inequalities and/or parametric restrictions. To overcome these limitations, we introduce a relaxed formulation that requires valid error control asymptotically with respect to a minimum sample size. This aligns with many real-world settings that often involve weak signals, high desired significance, and post-experiment inference requirements, all of which necessitate long horizons. This allows us to achieve tighter optimality, while better handling flexible nonparametric outcome distributions and fully leveraging individual-level contexts. We develop a novel asymptotic anytime-valid confidence sequences over arm indices, and we use it to design a new BAI algorithm for our asymptotic framework. Our method flexibly incorporates covariates for variance reduction and ensures approximate error control in fully nonparametric settings. Under mild convergence assumptions, we provide asymptotic bounds on the sample complexity and show the worst-case sample complexity of our approach matches the best-case sample complexity of Gaussian BAI under exact error guarantees and known variances. Experiments suggest our approach reduces average sample complexities while maintaining error control.

Exploration in the Limit

Abstract

In fixed-confidence best arm identification (BAI), the objective is to quickly identify the optimal option while controlling the probability of error below a desired threshold. Despite the plethora of BAI algorithms, existing methods typically fall short in practical settings, as stringent exact error control requires using loose tail inequalities and/or parametric restrictions. To overcome these limitations, we introduce a relaxed formulation that requires valid error control asymptotically with respect to a minimum sample size. This aligns with many real-world settings that often involve weak signals, high desired significance, and post-experiment inference requirements, all of which necessitate long horizons. This allows us to achieve tighter optimality, while better handling flexible nonparametric outcome distributions and fully leveraging individual-level contexts. We develop a novel asymptotic anytime-valid confidence sequences over arm indices, and we use it to design a new BAI algorithm for our asymptotic framework. Our method flexibly incorporates covariates for variance reduction and ensures approximate error control in fully nonparametric settings. Under mild convergence assumptions, we provide asymptotic bounds on the sample complexity and show the worst-case sample complexity of our approach matches the best-case sample complexity of Gaussian BAI under exact error guarantees and known variances. Experiments suggest our approach reduces average sample complexities while maintaining error control.
Paper Structure (70 sections, 30 theorems, 225 equations, 2 figures, 3 algorithms)

This paper contains 70 sections, 30 theorems, 225 equations, 2 figures, 3 algorithms.

Table of Contents

  1. Introduction
  2. Contributions
  3. Outline
  4. Related Works
  5. Best Arm Identification.
  6. Anytime-Valid Inference.
  7. Asymptotic Anytime-Valid Inference.
  8. Problem Formulation
  9. Modeling Assumptions
  10. Best Arm Identification
  11. Exploration with Confidence Sequences
  12. Constructing Test Processes
  13. Maximizing the Signal-to-Noise Ratio
  14. Information-Theoretic Interpretation
  15. Convex Reformulation for Optimization
  16. ...and 55 more sections

Key Result

Lemma 1

Assume that the context set $\mathcal{X}$ is empty and $a \neq a^*$. Let $\pi \in \Delta^K$ denote a vector on the $K$-dimensional probability simplex bounded away from zero. Let $d_{\sigma}(x,y) = \frac{(x-y)^2}{2\sigma^2}$ denote the KL divergence function between two Gaussian distributions with e

Figures (2)

  • Figure 1: Visualization of Confidence Sequence Approach. Solid lines plot score process $\hat{\psi}_t(a)$, and dotted lines plot asymptotic anytime-valid lower bounds $L_t^a(H_t, \alpha, \rho)$. Arm $a$ is removed from $C_t$ when $L_t^a(H_t, \alpha, \rho) > 0$.
  • Figure 2: Average number of samples under Bernoulli and Beta conditional outcome distributions. Error bars are $\pm1$ standard deviation for estimated average sample complexity over 100 simulations.

Theorems & Definitions (49)

  • Definition 1: $\alpha$-Correctness
  • Definition 2: Asymptotic $\alpha$-correctness
  • Remark 1: Choice of Index as a Burn-in Time
  • Definition 3: Score Process
  • Remark 2: Selection of $\rho$ Parameter
  • Lemma 1: SNR Maximization as KL-Projection
  • Remark 3: Connections to Testing-by-Betting.
  • Lemma 2: Charnes-Cooper-Schaible Transform
  • Theorem 1: Type I Error Control
  • Lemma 3: Asymptotically Valid BAI
  • ...and 39 more