Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Non-Stationary Bandit Learning via Predictive Sampling

Yueyang Liu, Xu Kuang, Benjamin Van Roy

TL;DR

This work tackles non-stationary bandit learning by arguing that standard Thompson sampling ignores information durability, which can lead to poor performance. It introduces predictive sampling (PS), which uses the sequence of future rewards as the learning target, guiding exploration toward more durable information. The authors develop a generalized information-theoretic regret framework based on predictive information and an information ratio, and provide regret bounds for PS, including for modulated Bernoulli and AR(1) bandits. They also offer tractable implementations and extensive experiments showing PS outperforming TS and other non-stationary approaches, with strong theoretical and empirical support for its effectiveness in dynamic environments.

Abstract

Thompson sampling has proven effective across a wide range of stationary bandit environments. However, as we demonstrate in this paper, it can perform poorly when applied to non-stationary environments. We attribute such failures to the fact that, when exploring, the algorithm does not differentiate actions based on how quickly the information acquired loses its usefulness due to non-stationarity. Building upon this insight, we propose predictive sampling, an algorithm that deprioritizes acquiring information that quickly loses usefulness. A theoretical guarantee on the performance of predictive sampling is established through a Bayesian regret bound. We provide versions of predictive sampling for which computations tractably scale to complex bandit environments of practical interest. Through numerical simulations, we demonstrate that predictive sampling outperforms Thompson sampling in all non-stationary environments examined.

Non-Stationary Bandit Learning via Predictive Sampling

TL;DR

This work tackles non-stationary bandit learning by arguing that standard Thompson sampling ignores information durability, which can lead to poor performance. It introduces predictive sampling (PS), which uses the sequence of future rewards as the learning target, guiding exploration toward more durable information. The authors develop a generalized information-theoretic regret framework based on predictive information and an information ratio, and provide regret bounds for PS, including for modulated Bernoulli and AR(1) bandits. They also offer tractable implementations and extensive experiments showing PS outperforming TS and other non-stationary approaches, with strong theoretical and empirical support for its effectiveness in dynamic environments.

Abstract

Thompson sampling has proven effective across a wide range of stationary bandit environments. However, as we demonstrate in this paper, it can perform poorly when applied to non-stationary environments. We attribute such failures to the fact that, when exploring, the algorithm does not differentiate actions based on how quickly the information acquired loses its usefulness due to non-stationarity. Building upon this insight, we propose predictive sampling, an algorithm that deprioritizes acquiring information that quickly loses usefulness. A theoretical guarantee on the performance of predictive sampling is established through a Bayesian regret bound. We provide versions of predictive sampling for which computations tractably scale to complex bandit environments of practical interest. Through numerical simulations, we demonstrate that predictive sampling outperforms Thompson sampling in all non-stationary environments examined.
Paper Structure (71 sections, 32 theorems, 121 equations, 11 figures, 5 algorithms)

This paper contains 71 sections, 32 theorems, 121 equations, 11 figures, 5 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Non-Stationary Bandit Learning
  4. Information-Theoretic Analysis of Stationary Bandit Learning
  5. Prediction Driven Decision Making
  6. Motivation
  7. Tossing Random Coins
  8. Existing Variants of TS Do Not Fix the Issue
  9. TS Can Perform Very Poorly
  10. Models and Definitions
  11. Stationary vs. Nonstationary Bandits
  12. Predictive Sampling
  13. Setting a Different Learning Target
  14. The Predictive Sampling Algorithm
  15. Qualitative Properties of PS
  16. ...and 56 more sections

Key Result

Theorem 1

For all $\epsilon \in (0, 1)$, there exists a Bernoulli bandit $\nu$ and a policy $\pi$ such that under $\nu$,

Figures (11)

  • Figure 1: Choosing between two coins: one with bias $p_1 = 0.99$ and the other with bias $p_{t,2}$ of either $1$ or $0$. The second coin is replaced each time with probability $0.99$.
  • Figure 2: Choosing among $K$ coins: the third through $K$-th coins are independent copies of the second coin.
  • Figure 3: Predictive sampling (PS)
  • Figure 4: The regret upper bound of PS ('ps:upr') and regret lower bounds of TS ('ts:lwr') and a random (uniform) policy ('random:lwr') in Example \ref{['ex:modulated_bernoulli_example']} with various values of $q_2$
  • Figure 5: The average rewards collected by PS and that collected by TS in AR(1) bandits
  • ...and 6 more figures

Theorems & Definitions (56)

  • Theorem 1
  • Example 1: Coin-Tossing Example Parameterized by $q$
  • Theorem 2
  • Example 2: Coin-Tossing Example with $K$ Coins
  • Proposition 1
  • Definition 1: Hindsight Regret
  • Proposition 2
  • Definition 2: Foresight Regret
  • Definition 3: Reversible Bandit
  • Proposition 3
  • ...and 46 more