Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

No-Regret Thompson Sampling for Finite-Horizon Markov Decision Processes with Gaussian Processes

Jasmine Bayrooti, Sattar Vakili, Amanda Prorok, Carl Henrik Ek

TL;DR

The paper addresses learning in finite-horizon episodic MDPs under model uncertainty by modeling rewards and transitions with a joint multi-output Gaussian Process and applying Thompson sampling (RL-GPS). It proves a no-regret bound of $\tilde{O}(\sqrt{KH\Gamma(KH)})$ and develops new tools, including confidence bounds for compositional GP outputs, a multi-output elliptical potential lemma, and a delayed-update analysis to handle batched updates. Theoretical results capture how kernel complexity via $\Gamma(T)$ governs learning efficiency, with explicit rates for Matérn and RBF kernels, and are supported by synthetic experiments showing sublinear regret and the benefits of output coupling via LMC. By extending GP analysis to vector-valued, recursive settings, the work strengthens the theoretical foundations of Thompson sampling for RL in continuous, structured environments and informs practical kernel choice for improved exploration.

Abstract

Thompson sampling (TS) is a powerful and widely used strategy for sequential decision-making, with applications ranging from Bayesian optimization to reinforcement learning (RL). Despite its success, the theoretical foundations of TS remain limited, particularly in settings with complex temporal structure such as RL. We address this gap by establishing no-regret guarantees for TS using models with Gaussian marginal distributions. Specifically, we consider TS in episodic RL with joint Gaussian process (GP) priors over rewards and transitions. We prove a regret bound of $\mathcal{\tilde{O}}(\sqrt{KHΓ(KH)})$ over $K$ episodes of horizon $H$, where $Γ(\cdot)$ captures the complexity of the GP model. Our analysis addresses several challenges, including the non-Gaussian nature of value functions and the recursive structure of Bellman updates, and extends classical tools such as the elliptical potential lemma to multi-output settings. This work advances the understanding of TS in RL and highlights how structural assumptions and model uncertainty shape its performance in finite-horizon Markov Decision Processes.

No-Regret Thompson Sampling for Finite-Horizon Markov Decision Processes with Gaussian Processes

TL;DR

The paper addresses learning in finite-horizon episodic MDPs under model uncertainty by modeling rewards and transitions with a joint multi-output Gaussian Process and applying Thompson sampling (RL-GPS). It proves a no-regret bound of and develops new tools, including confidence bounds for compositional GP outputs, a multi-output elliptical potential lemma, and a delayed-update analysis to handle batched updates. Theoretical results capture how kernel complexity via governs learning efficiency, with explicit rates for Matérn and RBF kernels, and are supported by synthetic experiments showing sublinear regret and the benefits of output coupling via LMC. By extending GP analysis to vector-valued, recursive settings, the work strengthens the theoretical foundations of Thompson sampling for RL in continuous, structured environments and informs practical kernel choice for improved exploration.

Abstract

Thompson sampling (TS) is a powerful and widely used strategy for sequential decision-making, with applications ranging from Bayesian optimization to reinforcement learning (RL). Despite its success, the theoretical foundations of TS remain limited, particularly in settings with complex temporal structure such as RL. We address this gap by establishing no-regret guarantees for TS using models with Gaussian marginal distributions. Specifically, we consider TS in episodic RL with joint Gaussian process (GP) priors over rewards and transitions. We prove a regret bound of over episodes of horizon , where captures the complexity of the GP model. Our analysis addresses several challenges, including the non-Gaussian nature of value functions and the recursive structure of Bellman updates, and extends classical tools such as the elliptical potential lemma to multi-output settings. This work advances the understanding of TS in RL and highlights how structural assumptions and model uncertainty shape its performance in finite-horizon Markov Decision Processes.
Paper Structure (31 sections, 6 theorems, 77 equations, 3 figures, 1 algorithm)

This paper contains 31 sections, 6 theorems, 77 equations, 3 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Contributions
  3. Related work
  4. RL with TS.
  5. Episodic MDP.
  6. Broader RL settings.
  7. Problem formulation
  8. Gaussian process modeling.
  9. Multi-output Gaussian processes.
  10. Reinforcement learning with GP sampling
  11. Analysis
  12. Confidence intervals
  13. Performance analysis of RL-GPS
  14. Proof of Theorem \ref{['the:regret_bound']}
  15. Elliptical potential lemma for multi-output GPs
  16. ...and 16 more sections

Key Result

Theorem 1

Assume $f:\mathcal{Z}\mapsto \mathcal{S}\subset \mathbb{R}^{d_{\text{S}}}$ is a multi-output GP with posterior mean $\mu_n$ and standard deviation $\sigma_n$. Let $v:\mathcal{S}\mapsto \mathbb{R}$ be a twice differentiable value function where for all $s \in \mathcal{S}$, $\|\nabla v(s)\| \le u_G$ a

Figures (3)

  • Figure 1: Cumulative regret over different kernels on GP-sampled environments over 200 trials. The shaded region around each curve represents $\pm$1 standard error of the mean across trials. Dotted lines represent median regrets.
  • Figure 2: The reward function (left column), optimal value function (middle column), and cumulative regret of TS with a multi-output GP with confidence regions representing the standard error over 200 random trials (right column) are given. The first row corresponds to the sparse navigation task and the second row corresponds to the sparse maze problem.
  • Figure 3: The first row shows the histogram of cumulative regrets over all trials on the last episode of RL-GPS training with each kernel. The second row shows the regret growth curves from all trials for each kernel.

Theorems & Definitions (12)

  • Remark 1
  • Theorem 1: Confidence bounds for composed GPs
  • Remark 2
  • Theorem 2
  • Remark 3
  • Remark 4
  • Definition 1: Upper and lower confidence bounds for value functions
  • Corollary 1
  • Lemma 1: Elliptical potential lemma for multi-output GPs
  • Lemma 2: Elliptical potential for multi-output GPs with delayed updates
  • ...and 2 more