On Dynamic Programming Decompositions of Static Risk Measures in Markov Decision Processes

TL;DR

The paper investigates dynamic-programming decompositions for static risk measures in finite-horizon MDPs, focusing on CVaR, EVaR, and VaR objectives. It shows that risk-level decompositions derived via dual representations for CVaR and EVaR are inherently suboptimal for policy optimization due to a saddle-point gap, although such decompositions can be valid for policy evaluation in certain cases. In contrast, the authors derive and validate a correct VaR-based DP that supports both policy evaluation and optimization, and provide a corrected EVaR decomposition for policy evaluation along with a counterexample demonstrating the inadequacy of Ni2022’s EVaR decomposition. The results highlight that current risk-averse DP methods used in high-stakes RL may be invalid or suboptimal, motivating more careful theoretical treatment and alternative approaches. Overall, the work clarifies which risk measures admit reliable DP decompositions and under what conditions, with direct implications for safety-critical decision-making systems.

Abstract

Optimizing static risk-averse objectives in Markov decision processes is difficult because they do not admit standard dynamic programming equations common in Reinforcement Learning (RL) algorithms. Dynamic programming decompositions that augment the state space with discrete risk levels have recently gained popularity in the RL community. Prior work has shown that these decompositions are optimal when the risk level is discretized sufficiently. However, we show that these popular decompositions for Conditional-Value-at-Risk (CVaR) and Entropic-Value-at-Risk (EVaR) are inherently suboptimal regardless of the discretization level. In particular, we show that a saddle point property assumed to hold in prior literature may be violated. However, a decomposition does hold for Value-at-Risk and our proof demonstrates how this risk measure differs from CVaR and EVaR. Our findings are significant because risk-averse algorithms are used in high-stake environments, making their correctness much more critical.

Figures (7)

• Figure 1: Rewards of MDP $M_{\mathrm{C}}$ used in the proof of \ref{['thm:cvar-wrong']}. The dot indicates that the rewards are independent of the next state.
• Figure 2: The functions $\theta_{\pi_1}(\cdot)$ and $\theta_{\pi_2}(\cdot)$ used in the CVaR counterexample in the proof of \ref{['thm:cvar-wrong']}. The dashed line shows the function $\zeta_{s_1} \mapsto \max_{\pi \in \left\{ \pi_1, \pi_2 \right\}} \theta_{\pi}([\zeta_{s_1}, 1-\zeta_{s_1}])$.
• Figure 3: Rewards of the MDP $M_{\mathrm{E}}$ used in the proof of \ref{['thm:evar-wrong']}. The dot indicates that the rewards are independent of the next state.
• Figure 4: An example used to show the sub-optimality of a policy in \ref{['sec:count-with-subopt']}.
• Figure 5: Return computed in Chow2015 vs optimal CVaR policy for the example in \ref{['fig:cvar-3actionscounterexample']} with $p_{s_1} = p_{s_2} = 0.5$ and $M = 600$. The optimal CVaR policy for each $\alpha$ is denoted by $\pi^\star$. The value function $\hat{v}$ is computed according to the decomposition in the r.h.s. of \ref{['eq:cvar-dec-optim']} and the corresponding policy is $\hat{\pi}$.

Theorems & Definitions (18)

• Proposition 3.1: lemma 22 in Pflug2016
• Theorem 3.2
• proof
• Theorem 4.1
• Theorem 4.2
• Example 1: Bernoulli random variable
• Theorem 5.1
• proof
• Theorem 5.2
• Proposition 5.3