Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VITS : Variational Inference Thompson Sampling for contextual bandits

Pierre Clavier, Tom Huix, Alain Durmus

TL;DR

This work tackles posterior intractability in Thompson Sampling for contextual bandits by introducing VITS, a Gaussian variational inference-based TS method. By using a non-degenerate Gaussian variational family and leveraging a Riemannian optimization framework, VITS provides tractable posterior samples and favorable computational properties. Theoretical results establish sub-linear regret in linear contextual bandits, and practical variants VITS-II and VITS-II Hessian-free reduce computational overhead while preserving performance. Empirical evaluations on synthetic benchmarks and the MovieLens dataset demonstrate competitive regret with real-world applicability and improved efficiency over traditional approximate TS methods. The approach offers a scalable, principled alternative for Bayesian exploration in contextual decision problems.

Abstract

In this paper, we introduce and analyze a variant of the Thompson sampling (TS) algorithm for contextual bandits. At each round, traditional TS requires samples from the current posterior distribution, which is usually intractable. To circumvent this issue, approximate inference techniques can be used and provide samples with distribution close to the posteriors. However, current approximate techniques yield to either poor estimation (Laplace approximation) or can be computationally expensive (MCMC methods, Ensemble sampling...). In this paper, we propose a new algorithm, Varational Inference Thompson sampling VITS, based on Gaussian Variational Inference. This scheme provides powerful posterior approximations which are easy to sample from, and is computationally efficient, making it an ideal choice for TS. In addition, we show that VITS achieves a sub-linear regret bound of the same order in the dimension and number of round as traditional TS for linear contextual bandit. Finally, we demonstrate experimentally the effectiveness of VITS on both synthetic and real world datasets.

VITS : Variational Inference Thompson Sampling for contextual bandits

TL;DR

This work tackles posterior intractability in Thompson Sampling for contextual bandits by introducing VITS, a Gaussian variational inference-based TS method. By using a non-degenerate Gaussian variational family and leveraging a Riemannian optimization framework, VITS provides tractable posterior samples and favorable computational properties. Theoretical results establish sub-linear regret in linear contextual bandits, and practical variants VITS-II and VITS-II Hessian-free reduce computational overhead while preserving performance. Empirical evaluations on synthetic benchmarks and the MovieLens dataset demonstrate competitive regret with real-world applicability and improved efficiency over traditional approximate TS methods. The approach offers a scalable, principled alternative for Bayesian exploration in contextual decision problems.

Abstract

In this paper, we introduce and analyze a variant of the Thompson sampling (TS) algorithm for contextual bandits. At each round, traditional TS requires samples from the current posterior distribution, which is usually intractable. To circumvent this issue, approximate inference techniques can be used and provide samples with distribution close to the posteriors. However, current approximate techniques yield to either poor estimation (Laplace approximation) or can be computationally expensive (MCMC methods, Ensemble sampling...). In this paper, we propose a new algorithm, Varational Inference Thompson sampling VITS, based on Gaussian Variational Inference. This scheme provides powerful posterior approximations which are easy to sample from, and is computationally efficient, making it an ideal choice for TS. In addition, we show that VITS achieves a sub-linear regret bound of the same order in the dimension and number of round as traditional TS for linear contextual bandit. Finally, we demonstrate experimentally the effectiveness of VITS on both synthetic and real world datasets.
Paper Structure (41 sections, 19 theorems, 175 equations, 7 figures, 3 tables, 3 algorithms)

This paper contains 41 sections, 19 theorems, 175 equations, 7 figures, 3 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related work.
  3. Notation.
  4. Thompson sampling for contextual bandits
  5. Contextual bandit:
  6. Thompson sampling:
  7. Variational inference TS:
  8. Presentation of $\textcolor{blue}{VITS-I}$:
  9. Presentation of VITS-II:
  10. Presentation of $\textcolor{purple}{VITS-II Hessian-free}$:
  11. Main results
  12. Linear Bandit
  13. Comparison table.
  14. Numerical experiments
  15. Linear and quadratic bandit
  16. ...and 26 more sections

Key Result

Theorem 3.5

Assume Assumptions as:subgaussian_rew to ass:prior hold. For the choice of hyperparameters $\{K_t,h_t\}_{t \in[T]}$ and $\eta$ specified in Section def:hyp, for any $\delta \in (0, 1)$, with probability at least $1-\delta$, the cumulative regret is bounded by

Figures (7)

  • Figure 1: Linear bandits, $\zeta=0.1$ (left), $\zeta=1$ (right).
  • Figure 2: Quadratic bandit, $\zeta=0.1$(left), $\zeta=1$(right).
  • Figure 3: Cumulative regret for MovieLens dataset.
  • Figure 4: Comparison Langevin Monte Carlo and Variational inference
  • Figure 5: Linear Bandits
  • ...and 2 more figures

Theorems & Definitions (36)

  • Theorem 3.5
  • Definition 1.1
  • Definition 1.2
  • Lemma 1.3
  • proof
  • Lemma 1.4
  • proof
  • Lemma 1.5
  • proof
  • Lemma 1.6
  • ...and 26 more