Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Posterior Sampling for Continuing Environments

Wanqiao Xu, Shi Dong, Benjamin Van Roy

TL;DR

The paper introduces Continuing PSRL, a posterior-sampling reinforcement learning method tailored for continuing environments, where the agent resamples a new environment model with probability $1-\gamma$ and plans using a $\gamma$-discounted objective. It proves a sublinear Bayesian regret bound $\tilde{O}(\tau S \sqrt{A T})$ and shows how a time-varying discount schedule $\gamma_t$ yields sublinear regret with respect to the optimal average reward, without requiring episodic resets. The analysis leverages a value-decomposition lemma and confidence sets to bound Bellman-error terms, achieving results comparable to previous TSDE-type approaches but with a simpler, scalable resampling mechanism. Empirical results on tabular and continuous RiverSwim variants demonstrate competitive performance and illustrate the practicality of resampling-based exploration in large or non-resetting environments, including fixes for function-approximation settings like bootstrapped DQN. Overall, the work clarifies the role of discounting in continuing RL and provides a scalable, theoretically-grounded exploration strategy for complex environments.

Abstract

We develop an extension of posterior sampling for reinforcement learning (PSRL) that is suited for a continuing agent-environment interface and integrates naturally into agent designs that scale to complex environments. The approach, continuing PSRL, maintains a statistically plausible model of the environment and follows a policy that maximizes expected $γ$-discounted return in that model. At each time, with probability $1-γ$, the model is replaced by a sample from the posterior distribution over environments. For a choice of discount factor that suitably depends on the horizon $T$, we establish an $\tilde{O}(τS \sqrt{A T})$ bound on the Bayesian regret, where $S$ is the number of environment states, $A$ is the number of actions, and $τ$ denotes the reward averaging time, which is a bound on the duration required to accurately estimate the average reward of any policy. Our work is the first to formalize and rigorously analyze the resampling approach with randomized exploration.

Posterior Sampling for Continuing Environments

TL;DR

The paper introduces Continuing PSRL, a posterior-sampling reinforcement learning method tailored for continuing environments, where the agent resamples a new environment model with probability and plans using a -discounted objective. It proves a sublinear Bayesian regret bound and shows how a time-varying discount schedule yields sublinear regret with respect to the optimal average reward, without requiring episodic resets. The analysis leverages a value-decomposition lemma and confidence sets to bound Bellman-error terms, achieving results comparable to previous TSDE-type approaches but with a simpler, scalable resampling mechanism. Empirical results on tabular and continuous RiverSwim variants demonstrate competitive performance and illustrate the practicality of resampling-based exploration in large or non-resetting environments, including fixes for function-approximation settings like bootstrapped DQN. Overall, the work clarifies the role of discounting in continuing RL and provides a scalable, theoretically-grounded exploration strategy for complex environments.

Abstract

We develop an extension of posterior sampling for reinforcement learning (PSRL) that is suited for a continuing agent-environment interface and integrates naturally into agent designs that scale to complex environments. The approach, continuing PSRL, maintains a statistically plausible model of the environment and follows a policy that maximizes expected -discounted return in that model. At each time, with probability , the model is replaced by a sample from the posterior distribution over environments. For a choice of discount factor that suitably depends on the horizon , we establish an bound on the Bayesian regret, where is the number of environment states, is the number of actions, and denotes the reward averaging time, which is a bound on the duration required to accurately estimate the average reward of any policy. Our work is the first to formalize and rigorously analyze the resampling approach with randomized exploration.
Paper Structure (17 sections, 7 theorems, 49 equations, 2 figures, 2 algorithms)

This paper contains 17 sections, 7 theorems, 49 equations, 2 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Continuing PSRL Algorithm
  4. The Algorithm
  5. Main Results
  6. Analysis
  7. Average Reward Regret
  8. Simulations
  9. Conclusion
  10. Bounding the sum of confidence set widths
  11. Continuing PSRL with $\gamma$-scheduling
  12. Proof of Theorem \ref{['thm:bayesian_regret']}
  13. Supporting Lemmas
  14. Proof of Lemma \ref{['lem:reward-averaging-time']}
  15. Proof of Lemma \ref{['lem:posterior_sampling']}
  16. ...and 2 more sections

Key Result

Lemma 3.3

For all $\mathcal{E}\in\Omega_*$, $s\in\mathcal{S}$, and $\gamma\in[0,1)$,

Figures (2)

  • Figure 1: Riverswim - continuous and dotted arrows represent the MDP under the actions "right" and "left", respectively.
  • Figure 2: Regret curves aggregated across 50 Monte Carlo runs for all algorithms.

Theorems & Definitions (17)

  • Definition 2.1
  • Definition 3.1
  • Lemma 3.3
  • Theorem 4.1
  • Theorem 4.2
  • Lemma 5.1
  • Lemma 5.2
  • proof
  • Lemma 5.3: Value decomposition
  • Lemma 5.4
  • ...and 7 more