Accelerated Online Risk-Averse Policy Evaluation in POMDPs with Theoretical Guarantees and Novel CVaR Bounds

TL;DR

This work introduces a theoretical framework for accelerating CVaR value function evaluation in POMDPs with formal performance guarantees, and develops estimators for these bounds within a particle-belief MDP framework with probabilistic guarantees.

Abstract

Risk-averse decision-making under uncertainty in partially observable domains is a central challenge in artificial intelligence and is essential for developing reliable autonomous agents. The formal framework for such problems is the partially observable Markov decision process (POMDP), where risk sensitivity is introduced through a risk measure applied to the value function, with Conditional Value-at-Risk (CVaR) being a particularly significant criterion. However, solving POMDPs is computationally intractable in general, and approximate methods rely on computationally expensive simulations of future agent trajectories. This work introduces a theoretical framework for accelerating CVaR value function evaluation in POMDPs with formal performance guarantees. We derive new bounds on the CVaR of a random variable X using an auxiliary random variable Y, under assumptions relating their cumulative distribution and density functions; these bounds yield interpretable concentration inequalities and converge as the distributional discrepancy vanishes. Building on this, we establish upper and lower bounds on the CVaR value function computable from a simplified belief-MDP, accommodating general simplifications of the transition dynamics. We develop estimators for these bounds within a particle-belief MDP framework with probabilistic guarantees, and employ them for acceleration via action elimination: actions whose bounds indicate suboptimality under the simplified model are safely discarded while ensuring consistency with the original POMDP. Empirical evaluation across multiple POMDP domains confirms that the bounds reliably separate safe from dangerous policies while achieving substantial computational speedups under the simplified model.

Figures (16)

• Figure 1: Overview of the proposed framework. The original POMDP yields the original CVaR value function. The simplified POMDP $M_s$ admits a simplified value function, related to the original via Theorem \ref{['thm:uniform_lower_and_upper_bounds_for_v_and_q']}. In practice, the simplified value function and distributional discrepancy are estimated from sample trajectories, yielding probabilistic computable bounds on the original value function (Theorem \ref{['thm:guarantees']}).
• Figure 2: Conceptual overview. Direct evaluation of $Q^\pi_M$ under the original model $M$ is computationally expensive. Instead, the simplified model $M_s$ is used to compute bounds on $Q^\pi_M$, where the bound quality is governed by the distributional discrepancy $\epsilon$ between their belief transition models.
• Figure 3: Problem formulation illustrations. (a) The CVaR value function focuses on the tail of the return distribution, unlike the expected value. (b) The simplified model $P_s$ yields a computable CDF $F_Y$, while the original CDF $F_X$ is intractable. When $|F_X(r) - F_Y(r)| \leq \epsilon$, bounds on $\mathop{\mathrm{CVaR}}\nolimits_\alpha(X)$ can be derived from $Y$.
• Figure 4: Illustrations of bounds on $F_X(x)$. (a) Bounds from Theorem \ref{['thm:cvar_bound_v2']}. (b) Bounds from Theorem \ref{['thm:tight_cvar_lower_bound']}, where $g$ depends on $x$ and yields a tighter result than $F_Y(x)+\epsilon$.
• Figure 5: Accumulation of distributional discrepancy over the planning horizon. Each bar represents the expected one-step TV distance $\mathbb{E}^s[\Delta^s(b_t, a_t) \mid b_k, a_k]$ at time $t$. The cumulative sum $\epsilon = \sum_{t=k}^{T-1} \Delta^s_t$ (dashed) governs the bound quality: when $\epsilon < \alpha$ (below the dash-dotted line), the CVaR bounds from Theorem \ref{['thm:uniform_lower_and_upper_bounds_for_v_and_q']} remain informative; when $\epsilon \geq \alpha$, the bounds reduce to trivial extrema.

Theorems & Definitions (63)

• Theorem 3.1
• Theorem 3.2
• Theorem 3.3
• Theorem 5.1
• proof
• Theorem 5.2
• proof
• Theorem 5.3
• proof
• Theorem 5.4