Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Distilled Thompson Sampling: Practical and Efficient Thompson Sampling via Imitation Learning

Hongseok Namkoong, Samuel Daulton, Eytan Bakshy

TL;DR

Distilled Thompson Sampling addresses the deployment bottleneck of contextual Thompson sampling by offline-distilling its policy into an explicit, fast policy representation via imitation learning. The approach preserves Bayes-regret performance close to batch on-policy TS, up to the cumulative imitation error, and provides generalization guarantees for imitation in contextual Gaussian process settings. Empirically, TS-IL matches batch-TS regret on benchmarks while reducing decision latency by orders of magnitude and delivering measurable gains in video quality and watch time in Meta deployments. This work enables practical, scalable Bayesian decision-making in latency-sensitive online platforms and invites extensions to broader policy classes and reinforcement learning.

Abstract

Thompson sampling (TS) has emerged as a robust technique for contextual bandit problems. However, TS requires posterior inference and optimization for action generation, prohibiting its use in many online platforms where latency and ease of deployment are of concern. We operationalize TS by proposing a novel imitation-learning-based algorithm that distills a TS policy into an explicit policy representation, allowing fast decision-making and easy deployment in mobile and server-based environments. Using batched data collected under the imitation policy, our algorithm iteratively performs offline updates to the TS policy, and learns a new explicit policy representation to imitate it. Empirically, our imitation policy achieves performance comparable to batch TS while allowing more than an order of magnitude reduction in decision-time latency. Buoyed by low latency and simplicity of implementation, our algorithm has been successfully deployed in multiple video upload systems for Meta. Using a randomized controlled trial, we show our algorithm resulted in significant improvements in video quality and watch time.

Distilled Thompson Sampling: Practical and Efficient Thompson Sampling via Imitation Learning

TL;DR

Distilled Thompson Sampling addresses the deployment bottleneck of contextual Thompson sampling by offline-distilling its policy into an explicit, fast policy representation via imitation learning. The approach preserves Bayes-regret performance close to batch on-policy TS, up to the cumulative imitation error, and provides generalization guarantees for imitation in contextual Gaussian process settings. Empirically, TS-IL matches batch-TS regret on benchmarks while reducing decision latency by orders of magnitude and delivering measurable gains in video quality and watch time in Meta deployments. This work enables practical, scalable Bayesian decision-making in latency-sensitive online platforms and invites extensions to broader policy classes and reinforcement learning.

Abstract

Thompson sampling (TS) has emerged as a robust technique for contextual bandit problems. However, TS requires posterior inference and optimization for action generation, prohibiting its use in many online platforms where latency and ease of deployment are of concern. We operationalize TS by proposing a novel imitation-learning-based algorithm that distills a TS policy into an explicit policy representation, allowing fast decision-making and easy deployment in mobile and server-based environments. Using batched data collected under the imitation policy, our algorithm iteratively performs offline updates to the TS policy, and learns a new explicit policy representation to imitate it. Empirically, our imitation policy achieves performance comparable to batch TS while allowing more than an order of magnitude reduction in decision-time latency. Buoyed by low latency and simplicity of implementation, our algorithm has been successfully deployed in multiple video upload systems for Meta. Using a randomized controlled trial, we show our algorithm resulted in significant improvements in video quality and watch time.

Paper Structure

This paper contains 33 sections, 11 theorems, 98 equations, 3 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Methodology
  3. Practical impact
  4. Theoretical contributions
  5. Related work
  6. Distilled Thompson sampling
  7. Empirical evaluation
  8. Datasets
  9. Algorithms and evaluation
  10. Imitation controls regret
  11. Regret decomposition
  12. Regret bounds for contextual Gaussian processes
  13. Bounds on $\gamma_T$
  14. Bounds on $\eta_{B}$
  15. Generalization guarantees for imitation learning
  16. ...and 18 more sections

Key Result

Lemma 1

Let $\{\pi_{\gamma(t)}\}_{t \in \mathbb{N}}$ and $U_t(\cdot; H_{\gamma(t)}, S_t)$ be any sequence of batch policies and UCBs (adapted to the history $H_{\gamma(t)}$). If $\mathbb{E}[\sup_{a \in \mathcal{A}} f_{\theta}(a, S)^2] =: L^2 < \infty$,

Figures (3)

  • Figure 1: An illustration of distilled TS on the example video uploads application, described in Example \ref{['example:video_transcoding']}. Online action generation is performed asynchronously on resource-constrained mobile edge devices whereas batched policy updates are performed offline on powerful backend servers.
  • Figure 2: We report mean cumulative regret (or running average of rewards for video transcoding), alongside two standard errors over 50 trials (100 trials for the Wheel bandit, due to rarity of large rewards).
  • Figure 3: Cumulative regret on the Warfarin problem with 50 actions

Theorems & Definitions (12)

  • Lemma 1
  • Theorem 1
  • Lemma 2
  • Proposition 2
  • Theorem 3
  • Corollary 1
  • Lemma 3
  • Lemma 4: SrinivasKrKaSe12
  • Lemma 5: BartlettBoMe05
  • Lemma 6
  • ...and 2 more