Thompson Sampling-like Algorithms for Stochastic Rising Bandits

TL;DR

This work addresses regret minimization for stochastic rising rested bandits by adapting Thompson Sampling with forced exploration and sliding windows. It introduces a complexity index σ_μ(T) and a new suboptimality gap notion, develops ET-Beta-SWTS and γ-ET-SWGTS with Beta/Gaussian posteriors, and proves instance-dependent regret bounds incorporating KL and TV-distance terms as the environment evolves. A pivotal technical lemma links Poisson-Binomial and Binomial distributions to bound underestimation of the optimal arm, enabling rigorous analysis in dynamic SRRBs. Lower bounds show the dependence on σ_μ(T) is intrinsic, and numerical experiments on synthetic and IMDB-inspired tasks confirm the practical efficacy of TS-like methods in regimes with manageable complexity. Overall, the paper advances our understanding of TS in non-stationary, rising environments and provides actionable guidance for deploying TS-like strategies in SRRB tasks.

Abstract

Stochastic rising rested bandit (SRRB) is a setting where the arms' expected rewards increase as they are pulled. It models scenarios in which the performances of the different options grow as an effect of an underlying learning process (e.g., online model selection). Even if the bandit literature provides specifically crafted algorithms based on upper-confidence bounds for such a setting, no study about Thompson sampling TS-like algorithms has been performed so far. The strong regularity of the expected rewards in the SRRB setting suggests that specific instances may be tackled effectively using adapted and sliding-window TS approaches. This work provides novel regret analyses for such algorithms in SRRBs, highlighting the challenges and providing new technical tools of independent interest. Our results allow us to identify under which assumptions TS-like algorithms succeed in achieving sublinear regret and which properties of the environment govern the complexity of the regret minimization problem when approached with TS. Furthermore, we provide a regret lower bound based on a complexity index we introduce. Finally, we conduct numerical simulations comparing TS-like algorithms with state-of-the-art approaches for SRRBs in synthetic and real-world settings.

Figures (13)

• Figure 1: 15-arm setting: expected reward functions of the arms.
• Figure 2: Average cumulative regret in the 15-arm setting ($50$ runs $\pm$ std).
• Figure 3: Average cumulative regret of (a) Beta-SWTS and $\gamma$-SWGTS for different window sizes $\tau=T^{\alpha}$ (b) ET-Beta-SWTS and $\gamma$-ET-SWGTS for different forced exploration $\Gamma$.
• Figure 4: Average cumulative regret in the IMBD setting ($30$ runs $\pm$ std).
• Figure 5: Different environments for the SRRB problem.

Theorems & Definitions (49)

• Definition 5.0
• Lemma 1: Expected Number of Pulls Bound for ET-Beta-SWTS
• Lemma 2: PB-Bin Stochastic Dominance
• Lemma 3: Wald's Inequality for Rising Bandits
• Theorem 6.1: ET-Beta-TS Bound
• Theorem 6.2: $\gamma$-ET-GTS Bound
• Corollary 1: Explicit Beta-TS and $\gamma$-GTS Bound
• Theorem 6.3: Lower Bound
• Theorem A.1: Theorem 4.4, metelli2022stochastic
• Theorem B.1: ET-Beta-SWTS Bound