Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Simplification of Risk Averse POMDPs with Performance Guarantees

Yaacov Pariente, Vadim Indelman

TL;DR

This work addresses risk-averse planning in partially observable domains by optimizing the CVaR of the return in POMDPs. It introduces a general simplification framework that substitutes a costly belief-MDP transition with a cheaper one and derives probabilistic performance guarantees for the resulting CVaR evaluation. Central contributions include theoretical bounds for CVaR via CDF/PDF differences, explicit bounds for the V and Q functions under model simplification, and practical estimators with finite-sample guarantees based on a Particle Belief MDP and importance sampling. The framework supports simultaneous simplification of observation and state models and aims to enable faster online planning for reliable autonomous agents while preserving safety guarantees, albeit with computational challenges in estimating some auxiliary quantities.

Abstract

Risk averse decision making under uncertainty in partially observable domains is a fundamental problem in AI and essential for reliable autonomous agents. In our case, the problem is modeled using partially observable Markov decision processes (POMDPs), when the value function is the conditional value at risk (CVaR) of the return. Calculating an optimal solution for POMDPs is computationally intractable in general. In this work we develop a simplification framework to speedup the evaluation of the value function, while providing performance guarantees. We consider as simplification a computationally cheaper belief-MDP transition model, that can correspond, e.g., to cheaper observation or transition models. Our contributions include general bounds for CVaR that allow bounding the CVaR of a random variable X, using a random variable Y, by assuming bounds between their cumulative distributions. We then derive bounds for the CVaR value function in a POMDP setting, and show how to bound the value function using the computationally cheaper belief-MDP transition model and without accessing the computationally expensive model in real-time. Then, we provide theoretical performance guarantees for the estimated bounds. Our results apply for a general simplification of a belief-MDP transition model and support simplification of both the observation and state transition models simultaneously.

Simplification of Risk Averse POMDPs with Performance Guarantees

TL;DR

This work addresses risk-averse planning in partially observable domains by optimizing the CVaR of the return in POMDPs. It introduces a general simplification framework that substitutes a costly belief-MDP transition with a cheaper one and derives probabilistic performance guarantees for the resulting CVaR evaluation. Central contributions include theoretical bounds for CVaR via CDF/PDF differences, explicit bounds for the V and Q functions under model simplification, and practical estimators with finite-sample guarantees based on a Particle Belief MDP and importance sampling. The framework supports simultaneous simplification of observation and state models and aims to enable faster online planning for reliable autonomous agents while preserving safety guarantees, albeit with computational challenges in estimating some auxiliary quantities.

Abstract

Risk averse decision making under uncertainty in partially observable domains is a fundamental problem in AI and essential for reliable autonomous agents. In our case, the problem is modeled using partially observable Markov decision processes (POMDPs), when the value function is the conditional value at risk (CVaR) of the return. Calculating an optimal solution for POMDPs is computationally intractable in general. In this work we develop a simplification framework to speedup the evaluation of the value function, while providing performance guarantees. We consider as simplification a computationally cheaper belief-MDP transition model, that can correspond, e.g., to cheaper observation or transition models. Our contributions include general bounds for CVaR that allow bounding the CVaR of a random variable X, using a random variable Y, by assuming bounds between their cumulative distributions. We then derive bounds for the CVaR value function in a POMDP setting, and show how to bound the value function using the computationally cheaper belief-MDP transition model and without accessing the computationally expensive model in real-time. Then, we provide theoretical performance guarantees for the estimated bounds. Our results apply for a general simplification of a belief-MDP transition model and support simplification of both the observation and state transition models simultaneously.
Paper Structure (24 sections, 32 theorems, 118 equations, 1 figure, 1 algorithm)

This paper contains 24 sections, 32 theorems, 118 equations, 1 figure, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Partially Observable Markov Decision Process
  4. Conditional Value-at-Risk
  5. Problem Formulation
  6. Bounds for the V and Q functions
  7. Theoretical CVaR bounds
  8. Theoretical value function bounds
  9. Bound estimation
  10. Q function estimator
  11. CDF bound estimation
  12. Performance guarantees
  13. Conclusions
  14. Acknowledgments
  15. Appendix
  16. ...and 9 more sections

Key Result

Theorem 2.1

If $\text{supp}(X) \subseteq [a, b]$ and $X$ has a continuous distribution function, then for any $\delta \in (0, 1]$,

Figures (1)

  • Figure 1: (a) and (b) are illustrations of the bounds on $F_X(x)$ from Theorems \ref{['thm:cvar_bound']} and \ref{['thm:tight_cvar_lower_bound']}. Since $g$ from Theorem \ref{['thm:tight_cvar_lower_bound']} depends on $x$, the bound $F_Y(x)+g(x)$ in (b) is tighter than $F_Y(x)+\epsilon$ in (a).

Theorems & Definitions (33)

  • Theorem 2.1
  • Theorem 3.1
  • Theorem 4.1
  • Theorem 4.2
  • Theorem 4.3
  • Theorem 4.4
  • Theorem 4.5
  • Theorem 4.6
  • Theorem 5.1
  • Theorem 5.2
  • ...and 23 more