Reward Redistribution for CVaR MDPs using a Bellman Operator on L-infinity

TL;DR

The paper tackles optimizing static CVaR in infinite-horizon MDPs, where the tail of the trajectory return $R(\tau)$ matters but recursive Bellman structure is absent. It introduces a transformed augmented MDP with a per-step reward $\tilde{r}$ and a CVaR-Bellman operator $\bar{T}$ that is a $\gamma$-contraction on the space of bounded functions, enabling tractable DP/TD learning and allowing discretization. By discretizing the augmented state budget $z$ and deriving tight upper and lower bounds via $\bar{T}^{\mathrm{l}}$, $\bar{T}^{\mathrm{u}}$, the approach enables risk-aware value iteration and Q-learning with convergence guarantees and error bounds. Empirical results on a stochastic gridworld show CVaR-sensitive policies and favorable performance-safety trade-offs, validating the theoretical guarantees. This framework lays the groundwork for scalable, risk-averse RL with function approximation beyond tabular settings.

Abstract

Tail-end risk measures such as static conditional value-at-risk (CVaR) are used in safety-critical applications to prevent rare, yet catastrophic events. Unlike risk-neutral objectives, the static CVaR of the return depends on entire trajectories without admitting a recursive Bellman decomposition in the underlying Markov decision process. A classical resolution relies on state augmentation with a continuous variable. However, unless restricted to a specialized class of admissible value functions, this formulation induces sparse rewards and degenerate fixed points. In this work, we propose a novel formulation of the static CVaR objective based on augmentation. Our alternative approach leads to a Bellman operator with: (1) dense per-step rewards; (2) contracting properties on the full space of bounded value functions. Building on this theoretical foundation, we develop risk-averse value iteration and model-free Q-learning algorithms that rely on discretized augmented states. We further provide convergence guarantees and approximation error bounds due to discretization. Empirical results demonstrate that our algorithms successfully learn CVaR-sensitive policies and achieve effective performance-safety trade-offs.

Figures (5)

• Figure 1: Illustration of the reward schemes produced along a trajectory under the augmented MDP of bauerle2011markov versus ours. (Per-step rewards shown in blue. Further details in Appendix \ref{['Appendix:mdp_variants']}).
• Figure 2: Stochastic Gridworld Environment.
• Figure 3: Performance comparison between the proposed static CVaR Q-learning and Q-VI algorithms for various risk-levels $\alpha$.
• Figure 4: Comparing the effect of augmented state discretization resolution on the accuracy of CVaR performance. As resolution increases (left to right) the performance curves under the upper and lower bounding Bellman operators converge to the true value.
• Figure 5: Illustration of the reward schemes produced along a trajectory under the augmented MDP of bauerle2011markov versus ours.

Theorems & Definitions (34)

• Proposition 1
• Lemma 2
• Proposition 3
• Theorem 4
• Proposition 5
• Theorem 6
• Theorem 7
• Proposition 8
• Proposition 9
• Theorem 10