Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Regenerative Particle Thompson Sampling

Zeyu Zhou, Bruce Hajek, Nakjung Choi, Anwar Walid

TL;DR

This work addresses the practical challenge of implementing Bayesian bandit strategies when the posterior is intractable by introducing Regenerative Particle Thompson Sampling (RPTS). Building on Particle Thompson Sampling (PTS), it provides a drift-based sample-path analysis showing that only a small subset of particles tends to survive, and unused particles decay; RPTS periodically regrows particles near surviving ones to sustain performance. Empirically, RPTS outperforms PTS across representative bandit problems and demonstrates efficacy in a network slicing application for 5G, highlighting its flexibility and practicality. The paper contributes a simple regeneration mechanism with intuitive hyperparameters and offers a path toward broader application in complex, structured bandit settings.

Abstract

This paper proposes regenerative particle Thompson sampling (RPTS), a flexible variation of Thompson sampling. Thompson sampling itself is a Bayesian heuristic for solving stochastic bandit problems, but it is hard to implement in practice due to the intractability of maintaining a continuous posterior distribution. Particle Thompson sampling (PTS) is an approximation of Thompson sampling obtained by simply replacing the continuous distribution by a discrete distribution supported at a set of weighted static particles. We observe that in PTS, the weights of all but a few fit particles converge to zero. RPTS is based on the heuristic: delete the decaying unfit particles and regenerate new particles in the vicinity of fit surviving particles. Empirical evidence shows uniform improvement from PTS to RPTS and flexibility and efficacy of RPTS across a set of representative bandit problems, including an application to 5G network slicing.

Regenerative Particle Thompson Sampling

TL;DR

This work addresses the practical challenge of implementing Bayesian bandit strategies when the posterior is intractable by introducing Regenerative Particle Thompson Sampling (RPTS). Building on Particle Thompson Sampling (PTS), it provides a drift-based sample-path analysis showing that only a small subset of particles tends to survive, and unused particles decay; RPTS periodically regrows particles near surviving ones to sustain performance. Empirically, RPTS outperforms PTS across representative bandit problems and demonstrates efficacy in a network slicing application for 5G, highlighting its flexibility and practicality. The paper contributes a simple regeneration mechanism with intuitive hyperparameters and offers a path toward broader application in complex, structured bandit settings.

Abstract

This paper proposes regenerative particle Thompson sampling (RPTS), a flexible variation of Thompson sampling. Thompson sampling itself is a Bayesian heuristic for solving stochastic bandit problems, but it is hard to implement in practice due to the intractability of maintaining a continuous posterior distribution. Particle Thompson sampling (PTS) is an approximation of Thompson sampling obtained by simply replacing the continuous distribution by a discrete distribution supported at a set of weighted static particles. We observe that in PTS, the weights of all but a few fit particles converge to zero. RPTS is based on the heuristic: delete the decaying unfit particles and regenerate new particles in the vicinity of fit surviving particles. Empirical evidence shows uniform improvement from PTS to RPTS and flexibility and efficacy of RPTS across a set of representative bandit problems, including an application to 5G network slicing.
Paper Structure (28 sections, 23 theorems, 83 equations, 18 figures, 5 algorithms)

This paper contains 28 sections, 23 theorems, 83 equations, 18 figures, 5 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Setup and Preliminaries
  4. Stochastic bandit problem
  5. Particle Thompson sampling (PTS)
  6. A Sample-Path Analysis of PTS
  7. RPTS: Regenerative Particle Thompson Sampling
  8. Application to Network Slicing
  9. Model
  10. Algorithm
  11. Simulation
  12. Conclusions and Future Work
  13. Proofs of Proposition \ref{['proposition:sample_path_necessary_survival_condition']} and Corollary \ref{['corollary:PTS_for_K_arm_Bern_bandit|number_of_surviving_particles_is_no_more_than_K']}
  14. Analysis of PTS for Two-Arm Bernoulli Bandit
  15. N given particles, weight dynamics
  16. ...and 13 more sections

Key Result

Proposition 1

Let $\mathrm{StochasticBandit}(\mathcal{A}, \Theta, \mathcal{Y}, P_\theta(\cdot|a), R, \theta^*)$ be a given problem and $\mathcal{P}_N \subset \Theta$ a given set of $N$ particles. Suppose $P_{\theta}(\cdot | a)$ satisfies Assumption assumption:PTS_for_general_stochastic_bandit|sample_path_asymptot where $S$ is the set in Assumption assumption:PTS_for_general_stochastic_bandit|sample_path_asympto

Figures (18)

  • Figure 1: Simulations
  • Figure 2: A network slicing model.
  • Figure 3: Per-block particles implementation.
  • Figure 4: Simulation for network slicing.
  • Figure 5: A divergence diagram example.
  • ...and 13 more figures

Theorems & Definitions (58)

  • Example 1: Bernoulli bandit
  • Example 2: Max-Bernoulli bandit
  • Example 3: Linear bandit
  • Definition 1
  • Proposition 1: Sample-path necessary surviving condition
  • Corollary 2
  • Lemma 3
  • proof
  • Lemma 4
  • proof
  • ...and 48 more