Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Information-Theoretic Analysis of Thompson Sampling with Infinite Action Spaces

Amaury Gouverneur, Borja Rodriguez Gálvez, Tobias Oechtering, Mikael Skoglund

TL;DR

This work addresses the Bayesian regret of Thompson Sampling (TS) in bandit problems with infinite and continuous action spaces by extending the information-theoretic rate-distortion framework. It introduces a one-step compressed TS based on a statistic of the optimal action, $A^\star_\varepsilon$, and proves a bound of the form $\mathbb{E}[\text{Regret}(T)] \leq \varepsilon T + \sqrt{\tilde{\Gamma} T \;H(A^\star_\varepsilon)}$, under a bound on the average compressed information ratio. For Lipschitz reward models, the bound further scales with the action space's complexity via covering numbers, yielding $\mathbb{E}[\text{Regret}(T)] \leq \sqrt{\tilde{\Gamma} T \log\mathcal{N}(\mathcal{A},\rho,\epsilon)} + L\epsilon T$. In the linear-bandit setting, the results recover the near-optimal rate $\mathcal{O}(d\sqrt{T\log T})$ with improved constants, demonstrating the framework's strength in handling continuous actions and high-dimensional problems. This provides a principled way to quantify TS performance in large or continuous action spaces and highlights how action-space complexity governs regret, with potential extensions to other bandit feedback models.

Abstract

This paper studies the Bayesian regret of the Thompson Sampling algorithm for bandit problems, building on the information-theoretic framework introduced by Russo and Van Roy (2015). Specifically, it extends the rate-distortion analysis of Dong and Van Roy (2018), which provides near-optimal bounds for linear bandits. A limitation of these results is the assumption of a finite action space. We address this by extending the analysis to settings with infinite and continuous action spaces. Additionally, we specialize our results to bandit problems with expected rewards that are Lipschitz continuous with respect to the action space, deriving a regret bound that explicitly accounts for the complexity of the action space.

An Information-Theoretic Analysis of Thompson Sampling with Infinite Action Spaces

TL;DR

This work addresses the Bayesian regret of Thompson Sampling (TS) in bandit problems with infinite and continuous action spaces by extending the information-theoretic rate-distortion framework. It introduces a one-step compressed TS based on a statistic of the optimal action, , and proves a bound of the form , under a bound on the average compressed information ratio. For Lipschitz reward models, the bound further scales with the action space's complexity via covering numbers, yielding . In the linear-bandit setting, the results recover the near-optimal rate with improved constants, demonstrating the framework's strength in handling continuous actions and high-dimensional problems. This provides a principled way to quantify TS performance in large or continuous action spaces and highlights how action-space complexity governs regret, with potential extensions to other bandit feedback models.

Abstract

This paper studies the Bayesian regret of the Thompson Sampling algorithm for bandit problems, building on the information-theoretic framework introduced by Russo and Van Roy (2015). Specifically, it extends the rate-distortion analysis of Dong and Van Roy (2018), which provides near-optimal bounds for linear bandits. A limitation of these results is the assumption of a finite action space. We address this by extending the analysis to settings with infinite and continuous action spaces. Additionally, we specialize our results to bandit problems with expected rewards that are Lipschitz continuous with respect to the action space, deriving a regret bound that explicitly accounts for the complexity of the action space.

Paper Structure

This paper contains 10 sections, 6 theorems, 22 equations, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. General Notation
  4. Problem setup
  5. Thompson Sampling algorithm
  6. Notations Specific to Bandit Problems
  7. Information ratio and Compressed TS
  8. Main Results
  9. Application to linear bandit problems
  10. Conclusion

Key Result

Proposition 1

Consider a space $\mathcal{A}$, two functions $f: \mathcal{A} \to \mathbb{R}_+$ and $g: \mathcal{A} \to \mathbb{R}_+$, and a probability distribution $\mathbb{Q}$ on $\mathcal{A}$. Then, there exists a pair $(a_1, a_2) \in \mathcal{A}^2$ and a $q \in [0,1]$ such that and

Theorems & Definitions (12)

  • Proposition 1
  • proof
  • Proposition 2
  • proof
  • Theorem 1
  • proof
  • Corollary 1
  • proof
  • Proposition 3
  • proof
  • ...and 2 more