A Simple and Optimal Policy Design with Safety against Heavy-Tailed Risk for Stochastic Bandits

TL;DR

The paper tackles stochastic bandits by designing policies that simultaneously achieve worst-case optimal expected regret and light-tailed risk in the regret distribution. It shows that traditional instance-dependent consistency cannot yield light tails, but a revised confidence-bound approach with a tailored bonus achieves both objectives, including any-time and linear-bandit extensions. The key methodological innovations include a split-and-conquer analysis to reduce dependence on the number of arms and a robust bonus design that preserves optimal regret while ensuring exponential tail decay. Empirically, the proposed policies deliver tighter tail distributions and greater robustness to hyperparameter tuning and risk misspecification, with extensions to the linear setting confirming the broad applicability of the approach.

Abstract

We study the stochastic multi-armed bandit problem and design new policies that enjoy both worst-case optimality for expected regret and light-tailed risk for regret distribution. Specifically, our policy design (i) enjoys the worst-case optimality for the expected regret at order $O(\sqrt{KT\ln T})$ and (ii) has the worst-case tail probability of incurring a regret larger than any $x>0$ being upper bounded by $\exp(-Ω(x/\sqrt{KT}))$, a rate that we prove to be best achievable with respect to $T$ for all worst-case optimal policies. Our proposed policy achieves a delicate balance between doing more exploration at the beginning of the time horizon and doing more exploitation when approaching the end, compared to standard confidence-bound-based policies. We also enhance the policy design to accommodate the "any-time" setting where $T$ is unknown a priori, and prove equivalently desired policy performances as compared to the "fixed-time" setting with known $T$. Numerical experiments are conducted to illustrate the theoretical findings. We find that from a managerial perspective, our new policy design yields better tail distributions and is preferable than celebrated policies especially when (i) there is a risk of under-estimating the volatility profile, or (ii) there is a challenge of tuning policy hyper-parameters. We conclude by extending our proposed policy design to the stochastic linear bandit setting that leads to both worst-case optimality in terms of expected regret and light-tailed risk on the regret distribution.

Figures (6)

• Figure 2: Empirical distribution for the cumulative reward
• Figure 3: Empirical distribution for the cumulative reward
• Figure 4: Regret tail decaying rate. Left: $\ln\mathbb P(\hat{R}_\theta^\pi(T)>0.2T)/\ln T$ vs $T$. Right: $\ln\mathbb P(\hat{R}_\theta^\pi(T)>0.04T)/\sqrt{T}$ vs $T$.
• Figure 5: Regret tail decaying rate. Left: $\ln\mathbb P(\hat{R}_\theta^\pi(T)>0.2T)/\ln T$ vs $T$. Right: $\ln\mathbb P(\hat{R}_\theta^\pi(T)>0.04T)/\sqrt{T}$ vs $T$.
• Figure 6: Empirical distributions for the cumulative reward; $x$-axis normalized

Theorems & Definitions (10)

• Lemma 1
• Theorem 1
• Theorem 2
• Theorem 3
• Theorem 4
• Theorem 5
• Theorem 6
• Theorem 7
• Lemma 2
• Lemma 3