Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Value-at-Risk Constrained Policy Optimization

Rohan Tangri, Jan-Peter Calliess

TL;DR

This work tackles tail-risk in reinforcement learning by directly optimizing Value-at-Risk (VaR) constraints. It proposes VaR-CPO, which uses a one-sided Chebyshev bound to create a differentiable, moment-based surrogate for VaR, and augments the state to enable a Markovian decomposition of second-order cost moments. The method extends Constrained Policy Optimization with robust trust-region guarantees, including a worst-case bound on constraint violations and a recovery mechanism when the mean cost is infeasible. Empirically, VaR-CPO achieves zero constraint violations in feasible environments and outperforms baselines on tail-risk heavy tasks like IcyLake and EcoAnt, demonstrating safe exploration without failures and a principled approach to deploying risk-sensitive RL in safety-critical domains.

Abstract

We introduce the Value-at-Risk Constrained Policy Optimization algorithm (VaR-CPO), a sample efficient and conservative method designed to optimize Value-at-Risk (VaR) constraints directly. Empirically, we demonstrate that VaR-CPO is capable of safe exploration, achieving zero constraint violations during training in feasible environments, a critical property that baseline methods fail to uphold. To overcome the inherent non-differentiability of the VaR constraint, we employ the one-sided Chebyshev inequality to obtain a tractable surrogate based on the first two moments of the cost return. Additionally, by extending the trust-region framework of the Constrained Policy Optimization (CPO) method, we provide rigorous worst-case bounds for both policy improvement and constraint violation during the training process.

Value-at-Risk Constrained Policy Optimization

TL;DR

This work tackles tail-risk in reinforcement learning by directly optimizing Value-at-Risk (VaR) constraints. It proposes VaR-CPO, which uses a one-sided Chebyshev bound to create a differentiable, moment-based surrogate for VaR, and augments the state to enable a Markovian decomposition of second-order cost moments. The method extends Constrained Policy Optimization with robust trust-region guarantees, including a worst-case bound on constraint violations and a recovery mechanism when the mean cost is infeasible. Empirically, VaR-CPO achieves zero constraint violations in feasible environments and outperforms baselines on tail-risk heavy tasks like IcyLake and EcoAnt, demonstrating safe exploration without failures and a principled approach to deploying risk-sensitive RL in safety-critical domains.

Abstract

We introduce the Value-at-Risk Constrained Policy Optimization algorithm (VaR-CPO), a sample efficient and conservative method designed to optimize Value-at-Risk (VaR) constraints directly. Empirically, we demonstrate that VaR-CPO is capable of safe exploration, achieving zero constraint violations during training in feasible environments, a critical property that baseline methods fail to uphold. To overcome the inherent non-differentiability of the VaR constraint, we employ the one-sided Chebyshev inequality to obtain a tractable surrogate based on the first two moments of the cost return. Additionally, by extending the trust-region framework of the Constrained Policy Optimization (CPO) method, we provide rigorous worst-case bounds for both policy improvement and constraint violation during the training process.
Paper Structure (23 sections, 1 theorem, 48 equations, 4 figures, 1 algorithm)

This paper contains 23 sections, 1 theorem, 48 equations, 4 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Constrained Markov Decision Process
  5. Constrained Policy Optimization
  6. Method
  7. Chebyshev Surrogate for Value-at-Risk
  8. Augmented Cost Formulation
  9. Update Step
  10. Worst-case Violation Bounds
  11. Cost Return Constraint
  12. Practical Algorithm
  13. Results
  14. Experimental Setup
  15. IcyLake Performance Analysis
  16. ...and 8 more sections

Key Result

Theorem 4.1

(Worst-Case Chebyshev Violation) For a policy update $\pi_{k+1}$ derived under the trust region constraint $\bar{D}_{KL}(\pi_k, \pi_{k+1}) \le \delta$, the true Chebyshev constraint violation is bounded by: where $\alpha^{\tilde{C}}_\pi = \max_s|\mathbb{E}_{a\sim \pi}[A^{\tilde{C}}_{\pi_k}(s, a)]|$ and $\alpha_\pi^C = \max_s|\mathbb{E}_{a\sim \pi}[A^C_{\pi_k}(s, a)]|$ represent the maximum expect

Figures (4)

  • Figure 1: Conservative Chebyshev Surrogate: The feasible VaR regions for cost threshold $\rho = 100$ and violation probability $\epsilon = 0.05$. The Chebyshev surrogate is valid for any distribution with finite first and second moments, which requires it to be overly conservative compared to a scenario where the underlying cost distribution is known to be Gaussian for example.
  • Figure 2: IcyLake Performance Analysis: Comparison of VaR-CPO (blue), PPO (orange), CPO (green) and CPPO (red) over 1m simulation timesteps. Shaded areas represent one standard deviation across 5 seeds. Figure \ref{['fig:reward_return']} shows the first 10k timesteps to highlight reward return convergence.
  • Figure 3: Probability mass function of the IcyLake environment state costs
  • Figure 4: EcoAnt Performance Analysis: Comparison of VaR-CPO (blue), PPO (orange), CPO (green), and CPPO (red) across battery sizes 50 (left - agents start unsafe) and 500 (right - agents start safe). Charts (a-b) show results over 5m timesteps, while (c-d) show 10m timesteps to better illustrate constraint satisfaction rates. Shaded areas represent one standard deviation across 5 seeds.

Theorems & Definitions (1)

  • Theorem 4.1