Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fixed-Confidence Best Arm Identification with Decreasing Variance

Tamojeet Roychowdhury, Kota Srinivas Reddy, Krishna P Jagannathan, Sharayu Moharir

TL;DR

The paper tackles fixed-confidence best-arm identification in a Gaussian multi-armed bandit where arm variances decline over time, introducing a cost that combines identification time and sampling effort. It proposes two policies: WTCS for settings with a known suboptimality gap, and PS-WSE for the gap-free case, the latter employing periodic sampling and weighted empirical means to cope with decreasing variance and sampling costs. The authors prove fixed-confidence guarantees and derive cost bounds for both policies, and corroborate their theoretical results with simulations showing improvements over classical stationary-bandit policies. The work highlights a new sampling paradigm under nonstationary variance and a trade-off between waiting to reduce variance and incurring longer termination times, with practical implications for efficient best-arm identification in time-varying environments.

Abstract

We focus on the problem of best-arm identification in a stochastic multi-arm bandit with temporally decreasing variances for the arms' rewards. We model arm rewards as Gaussian random variables with fixed means and variances that decrease with time. The cost incurred by the learner is modeled as a weighted sum of the time needed by the learner to identify the best arm, and the number of samples of arms collected by the learner before termination. Under this cost function, there is an incentive for the learner to not sample arms in all rounds, especially in the initial rounds. On the other hand, not sampling increases the termination time of the learner, which also increases cost. This trade-off necessitates new sampling strategies. We propose two policies. The first policy has an initial wait period with no sampling followed by continuous sampling. The second policy samples periodically and uses a weighted average of the rewards observed to identify the best arm. We provide analytical guarantees on the performance of both policies and supplement our theoretical results with simulations which show that our polices outperform the state-of-the-art policies for the classical best arm identification problem.

Fixed-Confidence Best Arm Identification with Decreasing Variance

TL;DR

The paper tackles fixed-confidence best-arm identification in a Gaussian multi-armed bandit where arm variances decline over time, introducing a cost that combines identification time and sampling effort. It proposes two policies: WTCS for settings with a known suboptimality gap, and PS-WSE for the gap-free case, the latter employing periodic sampling and weighted empirical means to cope with decreasing variance and sampling costs. The authors prove fixed-confidence guarantees and derive cost bounds for both policies, and corroborate their theoretical results with simulations showing improvements over classical stationary-bandit policies. The work highlights a new sampling paradigm under nonstationary variance and a trade-off between waiting to reduce variance and incurring longer termination times, with practical implications for efficient best-arm identification in time-varying environments.

Abstract

We focus on the problem of best-arm identification in a stochastic multi-arm bandit with temporally decreasing variances for the arms' rewards. We model arm rewards as Gaussian random variables with fixed means and variances that decrease with time. The cost incurred by the learner is modeled as a weighted sum of the time needed by the learner to identify the best arm, and the number of samples of arms collected by the learner before termination. Under this cost function, there is an incentive for the learner to not sample arms in all rounds, especially in the initial rounds. On the other hand, not sampling increases the termination time of the learner, which also increases cost. This trade-off necessitates new sampling strategies. We propose two policies. The first policy has an initial wait period with no sampling followed by continuous sampling. The second policy samples periodically and uses a weighted average of the rewards observed to identify the best arm. We provide analytical guarantees on the performance of both policies and supplement our theoretical results with simulations which show that our polices outperform the state-of-the-art policies for the classical best arm identification problem.

Paper Structure

This paper contains 14 sections, 3 theorems, 16 equations, 2 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Analytical Challenges and Contributions
  3. Related Works
  4. Problem Setup and Preliminaries
  5. Algorithms and Performance Guarantees
  6. Known Sub-optimality Gap
  7. No Side-information about Arm Means
  8. Simulation Results
  9. Simulation Settings
  10. Results
  11. Proofs
  12. Proof Outline for Theorem \ref{['thm:WTCS']}
  13. Proof Outline for Theorem \ref{['thm:PS-WSC']}
  14. Conclusions and Future Work

Key Result

Theorem 1

For a given $\delta \in (0,1)$, WTCS $\in \Pi(\delta)$ with

Figures (2)

  • Figure 1: The three plots on the left are for Experiment 1 (varying sub-optimality gap) and the ones the right are for Experiment 4 (varying sampling cost $c$). We plot the number of samples, the stopping time, and the total cost incurred by all four candidate policies.
  • Figure 2: Cost as a function of the varying parameter for varying numbers of arms on the left and for varying $\sigma$ on the right for all four candidate policies.

Theorems & Definitions (3)

  • Theorem 1
  • Theorem 2
  • Lemma 1