Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dynamic Programming for Epistemic Uncertainty in Markov Decision Processes

Axel Benyamine, Julien Grand-Clément, Marek Petrik, Michael I. Jordan, Alain Durmus

TL;DR

This work develops a unified theory of MDPs under epistemic uncertainty by treating transition kernels as random and evaluating policy return with a risk measure. It extends the dynamic programming framework through the ambiguity-averse value $V^{\pi,\nu,\rho}$ and Bellman operators $T^{\pi,\nu,\rho}$ and $T^{\nu,\rho}$, proving contraction and monotonicity under suitable conditions. A key contribution is the complete characterization of law-invariant risk measures compatible with DP: for static kernels, $\rho$ must be $\operatorname{ess\,inf}$ or $\operatorname{ess\,sup}$, and for resampled kernels it can also be $\mathbb{E}^{\nu}$; with $W^1$-continuity, only $\mathbb{E}^{\nu}$ remains. The results illuminate when DP-based methods yield stationary optimal policies and where one must go beyond DP (e.g., augmented state spaces or nested risk measures) to capture broader risk criteria in AMDPs.

Abstract

In this paper, we propose a general theory of ambiguity-averse MDPs, which treats the uncertain transition probabilities as random variables and evaluates a policy via a risk measure applied to its random return. This ambiguity-averse MDP framework unifies several models of MDPs with epistemic uncertainty for specific choices of risk measures. We extend the concepts of value functions and Bellman operators to our setting. Based on these objects, we establish the consequences of dynamic programming principles in this framework (existence of stationary policies, value and policy iteration algorithms), and we completely characterize law-invariant risk measures compatible with dynamic programming. Our work draws connections among several variants of MDP models and fully delineates what is possible under the dynamic programming paradigm and which risk measures require leaving it.

Dynamic Programming for Epistemic Uncertainty in Markov Decision Processes

TL;DR

This work develops a unified theory of MDPs under epistemic uncertainty by treating transition kernels as random and evaluating policy return with a risk measure. It extends the dynamic programming framework through the ambiguity-averse value and Bellman operators and , proving contraction and monotonicity under suitable conditions. A key contribution is the complete characterization of law-invariant risk measures compatible with DP: for static kernels, must be or , and for resampled kernels it can also be ; with -continuity, only remains. The results illuminate when DP-based methods yield stationary optimal policies and where one must go beyond DP (e.g., augmented state spaces or nested risk measures) to capture broader risk criteria in AMDPs.

Abstract

In this paper, we propose a general theory of ambiguity-averse MDPs, which treats the uncertain transition probabilities as random variables and evaluates a policy via a risk measure applied to its random return. This ambiguity-averse MDP framework unifies several models of MDPs with epistemic uncertainty for specific choices of risk measures. We extend the concepts of value functions and Bellman operators to our setting. Based on these objects, we establish the consequences of dynamic programming principles in this framework (existence of stationary policies, value and policy iteration algorithms), and we completely characterize law-invariant risk measures compatible with dynamic programming. Our work draws connections among several variants of MDP models and fully delineates what is possible under the dynamic programming paradigm and which risk measures require leaving it.
Paper Structure (52 sections, 20 theorems, 88 equations, 2 figures, 2 tables, 2 algorithms)

This paper contains 52 sections, 20 theorems, 88 equations, 2 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Notations
  3. Preliminaries
  4. Nominal MDPs
  5. Risk measures
  6. Ambiguity-averse MDPs
  7. Definition, objectives and examples
  8. Examples and related work
  9. Dynamic programming
  10. Main Results
  11. Computing optimal policies
  12. Dynamic programming compatibility
  13. Discussion
  14. Main takeaways
  15. Limitations and assumptions
  16. ...and 37 more sections

Key Result

Proposition 3.7

Let $\rho$ be a monotone, translation-invariant risk measure. Then we have the following properties for the operator $T \in \{ T^{\pi,\nu,\rho},T^{\nu,\rho}\}$: Finally, we have the following attainability property: for any $v \in \mathbb{R}^{\mathcal{S}}$, there exists $\pi \in \Pi_{\sf S}$ such that $T^{\nu,\rho}v = T^{\pi,\nu,\rho}v$.

Figures (2)

  • Figure 1: Ambiguity-averse MDP with two states and one action. The edges $(s,a,s')$ are labeled with pairs $(P(s,a,s'),r(s,a,s'))$.
  • Figure 2: Ambiguity-averse MDP with $\mathcal{S}=\{{\sf Start},{\sf End}\}$, $\mathcal{A}=\{1\}$. The edges $(s,a,s')$ are labeled with pairs $(P(s,a,s'),r(s,a,s'))$.

Theorems & Definitions (49)

  • Definition 3.1
  • Definition 3.2
  • Definition 3.3
  • Example 3.4
  • Remark 3.5
  • Remark 3.6
  • Proposition 3.7
  • Example 3.8: Continued from Example \ref{['ex: E, static main body']}
  • Theorem 4.1
  • Proposition 4.2
  • ...and 39 more