Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bayesian learning of the optimal action-value function in a Markov decision process

Jiaqi Guo, Chon Wai Ho, Sumeetpal S. Singh

TL;DR

This work develops a principled Bayesian framework for learning the optimal action-value function $Q^*$ in infinite-horizon, undiscounted MDPs with absorbing states, by constructing a likelihood directly from Bellman optimality equations and performing posterior inference on a parametric $Q_ heta$. It addresses both deterministic and stochastic rewards, introducing an ABC-like relaxation to maintain tractable inference when rewards are deterministic, and employing adaptive sequential Monte Carlo (SMC) with MCMC mutations for online learning. The approach generalizes Thompson sampling to MDPs by sampling policies from the posterior over optimal deterministic policies, enabling data-efficient exploration demonstrated on the Deep Sea benchmark where posterior sampling improves exploration and reduces regret compared to baselines like PSRL and bootstrapped DQN. The framework offers interpretable uncertainty quantification around $Q^*$, highlights challenges from non-identifiability and non-goal recurrent states, and provides a practical online algorithm that adaptively tunes tolerances and sampling strategies to balance approximation accuracy and computational cost. This work lays groundwork for principled Bayesian RL with exact BOE-driven likelihoods and adaptive, particle-based inference, potentially guiding future theory and scalable implementations in complex MDPs.

Abstract

The Markov Decision Process (MDP) is a popular framework for sequential decision-making problems, and uncertainty quantification is an essential component of it to learn optimal decision-making strategies. In particular, a Bayesian framework is used to maintain beliefs about the optimal decisions and the unknown ingredients of the model, which are also to be learned from the data, such as the rewards and state dynamics. However, many existing Bayesian approaches for learning the optimal decision-making strategy are based on unrealistic modelling assumptions and utilise approximate inference techniques. This raises doubts whether the benefits of Bayesian uncertainty quantification are fully realised or can be relied upon. We focus on infinite-horizon and undiscounted MDPs, with finite state and action spaces, and a terminal state. We provide a full Bayesian framework, from modelling to inference to decision-making. For modelling, we introduce a likelihood function with minimal assumptions for learning the optimal action-value function based on Bellman's optimality equations, analyse its properties, and clarify connections to existing works. For deterministic rewards, the likelihood is degenerate and we introduce artificial observation noise to relax it, in a controlled manner, to facilitate more efficient Monte Carlo-based inference. For inference, we propose an adaptive sequential Monte Carlo algorithm to both sample from and adjust the sequence of relaxed posterior distributions. For decision-making, we choose actions using samples from the posterior distribution over the optimal strategies. While commonly done, we provide new insight that clearly shows that it is a generalisation of Thompson sampling from multi-arm bandit problems. Finally, we evaluate our framework on the Deep Sea benchmark problem and demonstrate the exploration benefits of posterior sampling in MDPs.

Bayesian learning of the optimal action-value function in a Markov decision process

TL;DR

This work develops a principled Bayesian framework for learning the optimal action-value function in infinite-horizon, undiscounted MDPs with absorbing states, by constructing a likelihood directly from Bellman optimality equations and performing posterior inference on a parametric . It addresses both deterministic and stochastic rewards, introducing an ABC-like relaxation to maintain tractable inference when rewards are deterministic, and employing adaptive sequential Monte Carlo (SMC) with MCMC mutations for online learning. The approach generalizes Thompson sampling to MDPs by sampling policies from the posterior over optimal deterministic policies, enabling data-efficient exploration demonstrated on the Deep Sea benchmark where posterior sampling improves exploration and reduces regret compared to baselines like PSRL and bootstrapped DQN. The framework offers interpretable uncertainty quantification around , highlights challenges from non-identifiability and non-goal recurrent states, and provides a practical online algorithm that adaptively tunes tolerances and sampling strategies to balance approximation accuracy and computational cost. This work lays groundwork for principled Bayesian RL with exact BOE-driven likelihoods and adaptive, particle-based inference, potentially guiding future theory and scalable implementations in complex MDPs.

Abstract

The Markov Decision Process (MDP) is a popular framework for sequential decision-making problems, and uncertainty quantification is an essential component of it to learn optimal decision-making strategies. In particular, a Bayesian framework is used to maintain beliefs about the optimal decisions and the unknown ingredients of the model, which are also to be learned from the data, such as the rewards and state dynamics. However, many existing Bayesian approaches for learning the optimal decision-making strategy are based on unrealistic modelling assumptions and utilise approximate inference techniques. This raises doubts whether the benefits of Bayesian uncertainty quantification are fully realised or can be relied upon. We focus on infinite-horizon and undiscounted MDPs, with finite state and action spaces, and a terminal state. We provide a full Bayesian framework, from modelling to inference to decision-making. For modelling, we introduce a likelihood function with minimal assumptions for learning the optimal action-value function based on Bellman's optimality equations, analyse its properties, and clarify connections to existing works. For deterministic rewards, the likelihood is degenerate and we introduce artificial observation noise to relax it, in a controlled manner, to facilitate more efficient Monte Carlo-based inference. For inference, we propose an adaptive sequential Monte Carlo algorithm to both sample from and adjust the sequence of relaxed posterior distributions. For decision-making, we choose actions using samples from the posterior distribution over the optimal strategies. While commonly done, we provide new insight that clearly shows that it is a generalisation of Thompson sampling from multi-arm bandit problems. Finally, we evaluate our framework on the Deep Sea benchmark problem and demonstrate the exploration benefits of posterior sampling in MDPs.
Paper Structure (53 sections, 10 theorems, 106 equations, 9 figures, 2 tables, 6 algorithms)

This paper contains 53 sections, 10 theorems, 106 equations, 9 figures, 2 tables, 6 algorithms.

Table of Contents

  1. Introduction
  2. Notations
  3. Preliminaries
  4. Introduction to MDPs
  5. Uniqueness of solutions to Bellman optimality equations
  6. Related work
  7. Bayesian learning
  8. Bayesian formulation of learning Q*
  9. A definition via Bellman optimality equations
  10. Deterministic rewards
  11. Stochastic rewards
  12. Discussions on the intractable expectation within the likelihood
  13. Theoretical form of posterior under tabular Qθ and Gaussian likelihood
  14. From Thompson sampling to posterior sampling for MDPs
  15. Thompson sampling for multi-armed bandits
  16. ...and 38 more sections

Key Result

Theorem 1

Suppose Assumption ass:boeunique holds. Furthermore, the rewards are deterministic, i.e. $p^R$ is a Dirac function. Denote the conditional random variable (which is deterministic) as $r(s,a):=R_t|S_t=s,A_t=a$. Let $\mathcal{B}_v^*$ be an operator on $\mathcal{S} \rightarrow \mathbb{R}$ such that for for all $s \in \mathcal{S}$. Then,

Figures (9)

  • Figure 1: A deterministic 2-state MDP with a non-goal recurrent state. Each edge is labelled as (action, reward).
  • Figure 2: A deterministic 5-state MDP with tractable posterior. Each edge is labelled as (action, reward).
  • Figure 3: Left: Contour plot of the posterior of Example \ref{['ex:2dmdp']} with complete dataset $\mathcal{D}_2$, Gaussian prior with $\sigma=10$, tolerance $\epsilon=2$. Right: Contour plot of the posterior for Example \ref{['ex:2dmdp']} using the partial dataset $\mathcal{D}_1$, a zero-mean Gaussian prior with $\sigma = 10$, and tolerance $\epsilon = 0.01$.
  • Figure 4: The marginal posterior of $\theta_1$ and $\theta_2$ respectively of Example \ref{['ex:2dmdp']} with complete dataset $\mathcal{D}_2$, Gaussian prior $\sigma=10$, and various tolerances.
  • Figure 5: Deep Sea illustration randomisedvaluefunction.
  • ...and 4 more figures

Theorems & Definitions (27)

  • Theorem 1: bertseka1991
  • Lemma 2
  • proof
  • Definition 1
  • Theorem 3
  • proof
  • Example 1
  • Theorem 4
  • proof
  • Example 2
  • ...and 17 more