Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Near-optimal Per-Action Regret Bounds for Sleeping Bandits

Quan Nguyen, Nishant A. Mehta

TL;DR

This work studies sleeping bandits in the fully adversarial setting, where the active arm set ${\mathbb{A}}_t$ and losses are chosen by an adversary. It develops two main approaches—SB-EXP3 (generalized EXP3/EXP3-IX) and FTARL (FTRL with Tsallis entropy)—to obtain near-optimal per-action regret bounds, including $O(\sqrt{TA\ln{K}})$ and $O(\sqrt{T\sqrt{AK}})$, with both pseudo-regret and high-probability guarantees. The authors extend these results to bandits with sleeping experts via SE-EXP4, deriving standard minimax and adaptive/ tracking regret bounds, and they present a per-action strongly adaptive lower bound showing a fundamental trade-off: no algorithm can simultaneously achieve minimax-optimal per-action regret and sublinear per-action regret for all arms. The results unify and extend several existing adaptive and tracking regret bounds for non-sleeping bandits through sleeping-bandit analyses and offer new bounds for confidence-regret settings, with potential practical impact on systems where arm availability is dynamic and adversarial.

Abstract

We derive near-optimal per-action regret bounds for sleeping bandits, in which both the sets of available arms and their losses in every round are chosen by an adversary. In a setting with $K$ total arms and at most $A$ available arms in each round over $T$ rounds, the best known upper bound is $O(K\sqrt{TA\ln{K}})$, obtained indirectly via minimizing internal sleeping regrets. Compared to the minimax $Ω(\sqrt{TA})$ lower bound, this upper bound contains an extra multiplicative factor of $K\ln{K}$. We address this gap by directly minimizing the per-action regret using generalized versions of EXP3, EXP3-IX and FTRL with Tsallis entropy, thereby obtaining near-optimal bounds of order $O(\sqrt{TA\ln{K}})$ and $O(\sqrt{T\sqrt{AK}})$. We extend our results to the setting of bandits with advice from sleeping experts, generalizing EXP4 along the way. This leads to new proofs for a number of existing adaptive and tracking regret bounds for standard non-sleeping bandits. Extending our results to the bandit version of experts that report their confidences leads to new bounds for the confidence regret that depends primarily on the sum of experts' confidences. We prove a lower bound, showing that for any minimax optimal algorithms, there exists an action whose regret is sublinear in $T$ but linear in the number of its active rounds.

Near-optimal Per-Action Regret Bounds for Sleeping Bandits

TL;DR

This work studies sleeping bandits in the fully adversarial setting, where the active arm set and losses are chosen by an adversary. It develops two main approaches—SB-EXP3 (generalized EXP3/EXP3-IX) and FTARL (FTRL with Tsallis entropy)—to obtain near-optimal per-action regret bounds, including and , with both pseudo-regret and high-probability guarantees. The authors extend these results to bandits with sleeping experts via SE-EXP4, deriving standard minimax and adaptive/ tracking regret bounds, and they present a per-action strongly adaptive lower bound showing a fundamental trade-off: no algorithm can simultaneously achieve minimax-optimal per-action regret and sublinear per-action regret for all arms. The results unify and extend several existing adaptive and tracking regret bounds for non-sleeping bandits through sleeping-bandit analyses and offer new bounds for confidence-regret settings, with potential practical impact on systems where arm availability is dynamic and adversarial.

Abstract

We derive near-optimal per-action regret bounds for sleeping bandits, in which both the sets of available arms and their losses in every round are chosen by an adversary. In a setting with total arms and at most available arms in each round over rounds, the best known upper bound is , obtained indirectly via minimizing internal sleeping regrets. Compared to the minimax lower bound, this upper bound contains an extra multiplicative factor of . We address this gap by directly minimizing the per-action regret using generalized versions of EXP3, EXP3-IX and FTRL with Tsallis entropy, thereby obtaining near-optimal bounds of order and . We extend our results to the setting of bandits with advice from sleeping experts, generalizing EXP4 along the way. This leads to new proofs for a number of existing adaptive and tracking regret bounds for standard non-sleeping bandits. Extending our results to the bandit version of experts that report their confidences leads to new bounds for the confidence regret that depends primarily on the sum of experts' confidences. We prove a lower bound, showing that for any minimax optimal algorithms, there exists an action whose regret is sublinear in but linear in the number of its active rounds.
Paper Structure (32 sections, 35 theorems, 153 equations, 2 figures, 1 table)

This paper contains 32 sections, 35 theorems, 153 equations, 2 figures, 1 table.

Table of Contents

  1. INTRODUCTION
  2. PRELIMINARIES
  3. NEAR-OPTIMAL REGRET UPPER BOUNDS
  4. Generalized EXP3 for Sleeping Bandits
  5. Generalized FTRL for Sleeping Bandits
  6. Bounds on Confidence Regret
  7. BANDITS WITH ADVICE FROM SLEEPING EXPERTS
  8. Generalized EXP4
  9. Adaptive and Tracking Bounds for Standard Adversarial Bandits
  10. A PER-ACTION STRONGLY ADAPTIVE LOWER BOUND
  11. CONCLUSION
  12. PROOFS FOR SECTION \ref{['sec:SB-EXP3']}
  13. Proof of Lemma \ref{['lemma:upperboundQt1Qt']}
  14. Bounding the Estimated Regret
  15. Proof of Theorem \ref{['thm:inexpectationregretbound']}
  16. ...and 17 more sections

Key Result

Theorem 2

With $\gamma = 0$, for any $\eta > 0$, Algorithm algo:SB-EXP3 guarantees Tuning $\eta$ leads to an $O\left(\sqrt{\ln(G_T)\sum_{t=1}^T A_t}\right)$ bound.

Figures (2)

  • Figure 1: Environment $V_0$. All arms have loss equal 1 when they are active
  • Figure 2: Environment $V_{k}$. Except for arm $k$ which has losses equal to $0$, all arms have losses equal to $1$ when they are active.

Theorems & Definitions (66)

  • Remark 1
  • Theorem 2
  • Theorem 3
  • Lemma 3
  • Remark 4
  • Theorem 5
  • Theorem 6
  • Theorem 7
  • Remark 8
  • Lemma 8
  • ...and 56 more