Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Online Risk-Averse Planning in POMDPs Using Iterated CVaR Value Function

Yaacov Pariente, Vadim Indelman

TL;DR

This work addresses risk-sensitive planning under partial observability by adopting Iterated CVaR (ICVaR) as a dynamic risk measure. It develops a policy-evaluation method with finite-time guarantees and extends three online planning algorithms—Sparse Sampling, POMCPOW, and PFT-DPW—to optimize the ICVaR of returns, introducing a risk parameter $\alpha$ that controls aversion. Theoretical results provide finite-time bounds for both policy evaluation and sparse sampling under tail-risk objectives, and the proposed ICVaR-based MCTS variants incorporate a specialized exploration strategy. Empirical results on LaserTag and LightDark show that ICVaR planners achieve lower tail risk than their risk-neutral counterparts, demonstrating practical safety gains in POMDP planning. Overall, the paper presents a first-of-its-kind online risk-averse planning framework for POMDPs with provable guarantees and demonstrable tail-risk improvements.

Abstract

We study risk-sensitive planning under partial observability using the dynamic risk measure Iterated Conditional Value-at-Risk (ICVaR). A policy evaluation algorithm for ICVaR is developed with finite-time performance guarantees that do not depend on the cardinality of the action space. Building on this foundation, three widely used online planning algorithms--Sparse Sampling, Particle Filter Trees with Double Progressive Widening (PFT-DPW), and Partially Observable Monte Carlo Planning with Observation Widening (POMCPOW)--are extended to optimize the ICVaR value function rather than the expectation of the return. Our formulations introduce a risk parameter $α$, where $α= 1$ recovers standard expectation-based planning and $α< 1$ induces increasing risk aversion. For ICVaR Sparse Sampling, we establish finite-time performance guarantees under the risk-sensitive objective, which further enable a novel exploration strategy tailored to ICVaR. Experiments on benchmark POMDP domains demonstrate that the proposed ICVaR planners achieve lower tail risk compared to their risk-neutral counterparts.

Online Risk-Averse Planning in POMDPs Using Iterated CVaR Value Function

TL;DR

This work addresses risk-sensitive planning under partial observability by adopting Iterated CVaR (ICVaR) as a dynamic risk measure. It develops a policy-evaluation method with finite-time guarantees and extends three online planning algorithms—Sparse Sampling, POMCPOW, and PFT-DPW—to optimize the ICVaR of returns, introducing a risk parameter that controls aversion. Theoretical results provide finite-time bounds for both policy evaluation and sparse sampling under tail-risk objectives, and the proposed ICVaR-based MCTS variants incorporate a specialized exploration strategy. Empirical results on LaserTag and LightDark show that ICVaR planners achieve lower tail risk than their risk-neutral counterparts, demonstrating practical safety gains in POMDP planning. Overall, the paper presents a first-of-its-kind online risk-averse planning framework for POMDPs with provable guarantees and demonstrable tail-risk improvements.

Abstract

We study risk-sensitive planning under partial observability using the dynamic risk measure Iterated Conditional Value-at-Risk (ICVaR). A policy evaluation algorithm for ICVaR is developed with finite-time performance guarantees that do not depend on the cardinality of the action space. Building on this foundation, three widely used online planning algorithms--Sparse Sampling, Particle Filter Trees with Double Progressive Widening (PFT-DPW), and Partially Observable Monte Carlo Planning with Observation Widening (POMCPOW)--are extended to optimize the ICVaR value function rather than the expectation of the return. Our formulations introduce a risk parameter , where recovers standard expectation-based planning and induces increasing risk aversion. For ICVaR Sparse Sampling, we establish finite-time performance guarantees under the risk-sensitive objective, which further enable a novel exploration strategy tailored to ICVaR. Experiments on benchmark POMDP domains demonstrate that the proposed ICVaR planners achieve lower tail risk compared to their risk-neutral counterparts.
Paper Structure (27 sections, 7 theorems, 105 equations, 4 figures, 1 table, 7 algorithms)

This paper contains 27 sections, 7 theorems, 105 equations, 4 figures, 1 table, 7 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Partially Observable Markov Decision Process
  4. Conditional Value at Risk
  5. Problem Formulation
  6. ICVaR Policy Evaluation
  7. ICVaR Sparse Sampling
  8. Performance Guarantees
  9. ICVaR Policy Evaluation Guarantees
  10. ICVaR Sparse Sampling Guarantees
  11. Remarks
  12. MCTS Algorithms
  13. ICVaR Exploration
  14. ICVaR-POMCPOW and ICVaR-PFT-DPW
  15. Experiments
  16. ...and 12 more sections

Key Result

Theorem 1

Let $\delta \in (0,1)$, and particle belief at time $t$ be $\bar{b}_t=\{x_t^i, w_t^i\}_{i=1}^{N_p}$. Define $\Delta R \triangleq R_{\max}-R_{\min}$, $T_{\alpha,t}\triangleq\sum_{k=0}^{T-t-1} \frac{T-t-k}{\alpha^k}$, and $T'_{\alpha,t}\triangleq\sum_{j=0}^{T-t-1} \frac{T-t+1-j}{\alpha^j}$. If $N_b>1$

Figures (4)

  • Figure 1: Search tree structure
  • Figure 2: ICVaR computation for action $a_0^1$ with $\alpha=0.01$
  • Figure 4: ICVaR policy evaluation belief tree with $N_b=4$.
  • Figure 5: ICVaR sparse sampling tree with value distributions at action nodes. Blue bars: non-tail region; red bars: upper $\alpha$-tail. Dashed line: VaR threshold; solid red line: CVaR.

Theorems & Definitions (13)

  • Theorem 1
  • proof
  • Theorem 2
  • proof : Proof sketch
  • Theorem 3
  • proof
  • Theorem 4
  • proof
  • Theorem 5
  • proof
  • ...and 3 more