Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Data-Driven Upper Confidence Bounds with Near-Optimal Regret for Heavy-Tailed Bandits

Ambrus Tamás, Szabolcs Szentpéteri, Balázs Csanád Csáji

TL;DR

The paper tackles heavy-tailed stochastic multi-armed bandits with symmetric reward distributions and unknown moment parameters. It introduces a data-driven, parameter-free upper confidence bound built on a one-sided resampled median-of-means estimator to produce exact bounds for the means. The main contributions include near-optimal regret bounds without knowing the tail parameter a or the moment M, exact confidence guarantees for the UCB, and showing that MARS is a special case of the proposed framework. Empirical results demonstrate robust performance and improved regret in difficult heavy-tailed settings, highlighting practical applicability in real-world bandit problems where moments are unavailable.

Abstract

Stochastic multi-armed bandits (MABs) provide a fundamental reinforcement learning model to study sequential decision making in uncertain environments. The upper confidence bounds (UCB) algorithm gave birth to the renaissance of bandit algorithms, as it achieves near-optimal regret rates under various moment assumptions. Up until recently most UCB methods relied on concentration inequalities leading to confidence bounds which depend on moment parameters, such as the variance proxy, that are usually unknown in practice. In this paper, we propose a new distribution-free, data-driven UCB algorithm for symmetric reward distributions, which needs no moment information. The key idea is to combine a refined, one-sided version of the recently developed resampled median-of-means (RMM) method with UCB. We prove a near-optimal regret bound for the proposed anytime, parameter-free RMM-UCB method, even for heavy-tailed distributions.

Data-Driven Upper Confidence Bounds with Near-Optimal Regret for Heavy-Tailed Bandits

TL;DR

The paper tackles heavy-tailed stochastic multi-armed bandits with symmetric reward distributions and unknown moment parameters. It introduces a data-driven, parameter-free upper confidence bound built on a one-sided resampled median-of-means estimator to produce exact bounds for the means. The main contributions include near-optimal regret bounds without knowing the tail parameter a or the moment M, exact confidence guarantees for the UCB, and showing that MARS is a special case of the proposed framework. Empirical results demonstrate robust performance and improved regret in difficult heavy-tailed settings, highlighting practical applicability in real-world bandit problems where moments are unavailable.

Abstract

Stochastic multi-armed bandits (MABs) provide a fundamental reinforcement learning model to study sequential decision making in uncertain environments. The upper confidence bounds (UCB) algorithm gave birth to the renaissance of bandit algorithms, as it achieves near-optimal regret rates under various moment assumptions. Up until recently most UCB methods relied on concentration inequalities leading to confidence bounds which depend on moment parameters, such as the variance proxy, that are usually unknown in practice. In this paper, we propose a new distribution-free, data-driven UCB algorithm for symmetric reward distributions, which needs no moment information. The key idea is to combine a refined, one-sided version of the recently developed resampled median-of-means (RMM) method with UCB. We prove a near-optimal regret bound for the proposed anytime, parameter-free RMM-UCB method, even for heavy-tailed distributions.
Paper Structure (20 sections, 15 theorems, 65 equations, 4 figures, 3 algorithms)

This paper contains 20 sections, 15 theorems, 65 equations, 4 figures, 3 algorithms.

Table of Contents

  1. Introduction
  2. Stochastic Multi-Armed Bandits
  3. Data-Driven Bandits
  4. Heavy-Tailed Bandits
  5. Median-of-Means Type Upper Confidence Bounds
  6. Remarks on the Empirical Mean
  7. Median-of-Means
  8. One-Sided Resampled Median-of-Means
  9. Resampled Median-of-Means Upper Confidence Bounds
  10. Resampled Median-of-Means UCB Algorithm
  11. Numerical Experiments
  12. Conclusion
  13. Proof of Theorem \ref{['thm:exact-coverage']}
  14. Proof of Lemma \ref{['lemma:finite-rep']}
  15. Proof of Theorem \ref{['thm:ucb']}
  16. ...and 5 more sections

Key Result

Theorem 2.1

Assume Aass:iid and Aass:finite_mom and let $\mathbb{E} [X_1] = \mu$, then the MoM estimator $\widehat{\mu}$ with $k = \lceil8 \log(2/\delta)\rceil$ blocks satisfies Moreover, for any mean estimator $\mu_n$, sample size $n \in \mathbb{N}$ and $\delta > 0$ there exists a distribution with mean $\mu$ and $(1+a)$th central moment $M$ such that

Figures (4)

  • Figure 1: Comparison of (average) cumulative regrets for Pareto bandits with $\varepsilon_p = 0.1$.
  • Figure 2: Comparison of average cumulative regrets for Pareto bandits, $\varepsilon_p = 0.5$.
  • Figure 3: Comparison of std. deviations of cumulative regrets for Pareto bandits, $\varepsilon_p = 0.5$.
  • Figure 4: Comparison of std. deviations of cumulative regrets for Pareto bandits, $\varepsilon_p = 0.1$.

Theorems & Definitions (20)

  • Theorem 2.1
  • Theorem 2.2
  • Corollary 2.2.1
  • Lemma 2.3
  • Theorem 2.4
  • Theorem 3.1
  • Definition A.1: ranking function
  • Lemma A.1
  • Theorem A.2
  • Lemma B.1
  • ...and 10 more