Risk-Averse Constrained Reinforcement Learning with Optimized Certainty Equivalents

TL;DR

This work tackles risk-aware decision-making in constrained reinforcement learning by employing optimized certainty equivalents (OCEs), including CVaR, to express risk in both objectives and constraints. It develops a parameterized strong duality framework with an exact partial Lagrangian relaxation, enabling a practical minimax formulation that can wrap around black-box RL solvers such as PPO. A convergence analysis validates the online algorithm under standard assumptions, and empirical results on Safety-Gym demonstrate strict constraint satisfaction and improved risk management compared to risk-neutral baselines. The approach offers a flexible, modular path to incorporating tail-risk considerations in high-stakes RL applications, with general applicability beyond CVaR to a broad class of OCEs, albeit with some computational overhead and remaining questions about unconditional duality.

Abstract

Constrained optimization provides a common framework for dealing with conflicting objectives in reinforcement learning (RL). In most of these settings, the objectives (and constraints) are expressed though the expected accumulated reward. However, this formulation neglects risky or even possibly catastrophic events at the tails of the reward distribution, and is often insufficient for high-stakes applications in which the risk involved in outliers is critical. In this work, we propose a framework for risk-aware constrained RL, which exhibits per-stage robustness properties jointly in reward values and time using optimized certainty equivalents (OCEs). Our framework ensures an exact equivalent to the original constrained problem within a parameterized strong Lagrangian duality framework under appropriate constraint qualifications, and yields a simple algorithmic recipe which can be wrapped around standard RL solvers, such as PPO. Lastly, we establish the convergence of the proposed algorithm under common assumptions, and verify the risk-aware properties of our approach through several numerical experiments.

Figures (1)

• Figure 1: Learning curves of the agent trained to convergence. We report episodic evaluation rewards, the progression of optimization variables $t$ and $\lambda$, and the velocity distribution of the trained agent evaluated over 100 post-training episodes without further learning. Curves are not smoothed.

Theorems & Definitions (19)

• Remark 3.1
• Lemma 3.2
• Proposition 3.3
• Theorem 3.5: Almost-zero Duality Gap
• Lemma 3.7
• Remark 3.9
• Definition 3.11: $\epsilon$-stationary point
• Theorem 3.12
• proof
• proof