Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MDP Planning as Policy Inference

David Tolpin

TL;DR

Across grid worlds, Blackjack, Triangle Tireworld, and Academic Advising, the structure of inferred policy distributions is analyzed and the resulting behavior to discrete Soft Actor-Critic is compared, highlighting qualitative and statistical differences that arise from policy-level uncertainty.

Abstract

We cast episodic Markov decision process (MDP) planning as Bayesian inference over _policies_. A policy is treated as the latent variable and is assigned an unnormalized probability of optimality that is monotone in its expected return, yielding a posterior distribution whose modes coincide with return-maximizing solutions while posterior dispersion represents uncertainty over optimal behavior. To approximate this posterior in discrete domains, we adapt variational sequential Monte Carlo (VSMC) to inference over deterministic policies under stochastic dynamics, introducing a sweep that enforces policy consistency across revisited states and couples transition randomness across particles to avoid confounding from simulator noise. Acting is performed by posterior predictive sampling, which induces a stochastic control policy through a Thompson-sampling interpretation rather than entropy regularization. Across grid worlds, Blackjack, Triangle Tireworld, and Academic Advising, we analyze the structure of inferred policy distributions and compare the resulting behavior to discrete Soft Actor-Critic, highlighting qualitative and statistical differences that arise from policy-level uncertainty.

MDP Planning as Policy Inference

TL;DR

Across grid worlds, Blackjack, Triangle Tireworld, and Academic Advising, the structure of inferred policy distributions is analyzed and the resulting behavior to discrete Soft Actor-Critic is compared, highlighting qualitative and statistical differences that arise from policy-level uncertainty.

Abstract

We cast episodic Markov decision process (MDP) planning as Bayesian inference over _policies_. A policy is treated as the latent variable and is assigned an unnormalized probability of optimality that is monotone in its expected return, yielding a posterior distribution whose modes coincide with return-maximizing solutions while posterior dispersion represents uncertainty over optimal behavior. To approximate this posterior in discrete domains, we adapt variational sequential Monte Carlo (VSMC) to inference over deterministic policies under stochastic dynamics, introducing a sweep that enforces policy consistency across revisited states and couples transition randomness across particles to avoid confounding from simulator noise. Acting is performed by posterior predictive sampling, which induces a stochastic control policy through a Thompson-sampling interpretation rather than entropy regularization. Across grid worlds, Blackjack, Triangle Tireworld, and Academic Advising, we analyze the structure of inferred policy distributions and compare the resulting behavior to discrete Soft Actor-Critic, highlighting qualitative and statistical differences that arise from policy-level uncertainty.
Paper Structure (29 sections, 1 theorem, 16 equations, 8 figures, 3 tables, 3 algorithms)

This paper contains 29 sections, 1 theorem, 16 equations, 8 figures, 3 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Markov Decision Process
  4. Variational Sequential Monte Carlo
  5. Probabilistic Model
  6. Inference
  7. Algorithm
  8. SMC sweep
  9. Optimization objective
  10. Policy Selection
  11. Deterministic Policy Extraction
  12. Annealing-Based MAP Selection
  13. Stochastic Policy Selection via Posterior Predictive Sampling
  14. Related Work
  15. Control and Planning as Inference
  16. ...and 14 more sections

Key Result

Theorem 1

Let $\hat{Z}(\hat{T}, \mathbf a)$ denote the SMC normalizing constant estimator produced by a sweep of the modified algorithm, where $\hat{T}$ is the shared random realization of the transition dynamics and $\mathbf a=\{a_{t,i}\}_{t=1,i=1}^{H,N}$ are the actions sampled from the proposal $q_\theta$. Then the gradient of the surrogate objective in Eqs. eqn:det-surrogate-objective and eqn:det-surrog

Figures (8)

  • Figure 1: Grid worlds: policies and occupancies
  • Figure 2: Grid worlds: VSMC variations on multimodal world
  • Figure 3: Grid worlds: deterministic policies not enforced
  • Figure 4: Grid worlds: independent environment dynamics
  • Figure 5: Grid worlds: VSMC vs. SAC
  • ...and 3 more figures

Theorems & Definitions (3)

  • Definition 1
  • Theorem 1: Unbiased gradient estimator
  • proof : Proof outline