Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Extreme Value Policy Optimization for Safe Reinforcement Learning

Shiqing Gao, Yihang Zhou, Shuai Shao, Haoyu Luo, Yiheng Bing, Jiaxin Ding, Luoyi Fu, Xinbing Wang

TL;DR

This work addresses tail-risk in constrained reinforcement learning by integrating Extreme Value Theory (EVT) into policy optimization. EVO builds an extreme quantile constraint via a Generalized Pareto Distribution (GPD) fitted to tail samples, and leverages extreme prioritization and off-policy resampling to robustly learn from rare but informative events. The authors prove upper bounds on expected constraint violations, derive a zero-violation exploitation range $\nu_0$, and show reduced violation probability and variance relative to baselines. Empirical results on Safety Gymnasium and Safety MuJoCo demonstrate significant reductions in constraint violations with competitive task performance, validating both the theoretical guarantees and practical utility of EVT-driven tail learning in safe RL.

Abstract

Ensuring safety is a critical challenge in applying Reinforcement Learning (RL) to real-world scenarios. Constrained Reinforcement Learning (CRL) addresses this by maximizing returns under predefined constraints, typically formulated as the expected cumulative cost. However, expectation-based constraints overlook rare but high-impact extreme value events in the tail distribution, such as black swan incidents, which can lead to severe constraint violations. To address this issue, we propose the Extreme Value policy Optimization (EVO) algorithm, leveraging Extreme Value Theory (EVT) to model and exploit extreme reward and cost samples, reducing constraint violations. EVO introduces an extreme quantile optimization objective to explicitly capture extreme samples in the cost tail distribution. Additionally, we propose an extreme prioritization mechanism during replay, amplifying the learning signal from rare but high-impact extreme samples. Theoretically, we establish upper bounds on expected constraint violations during policy updates, guaranteeing strict constraint satisfaction at a zero-violation quantile level. Further, we demonstrate that EVO achieves a lower probability of constraint violations than expectation-based methods and exhibits lower variance than quantile regression methods. Extensive experiments show that EVO significantly reduces constraint violations during training while maintaining competitive policy performance compared to baselines.

Extreme Value Policy Optimization for Safe Reinforcement Learning

TL;DR

This work addresses tail-risk in constrained reinforcement learning by integrating Extreme Value Theory (EVT) into policy optimization. EVO builds an extreme quantile constraint via a Generalized Pareto Distribution (GPD) fitted to tail samples, and leverages extreme prioritization and off-policy resampling to robustly learn from rare but informative events. The authors prove upper bounds on expected constraint violations, derive a zero-violation exploitation range , and show reduced violation probability and variance relative to baselines. Empirical results on Safety Gymnasium and Safety MuJoCo demonstrate significant reductions in constraint violations with competitive task performance, validating both the theoretical guarantees and practical utility of EVT-driven tail learning in safe RL.

Abstract

Ensuring safety is a critical challenge in applying Reinforcement Learning (RL) to real-world scenarios. Constrained Reinforcement Learning (CRL) addresses this by maximizing returns under predefined constraints, typically formulated as the expected cumulative cost. However, expectation-based constraints overlook rare but high-impact extreme value events in the tail distribution, such as black swan incidents, which can lead to severe constraint violations. To address this issue, we propose the Extreme Value policy Optimization (EVO) algorithm, leveraging Extreme Value Theory (EVT) to model and exploit extreme reward and cost samples, reducing constraint violations. EVO introduces an extreme quantile optimization objective to explicitly capture extreme samples in the cost tail distribution. Additionally, we propose an extreme prioritization mechanism during replay, amplifying the learning signal from rare but high-impact extreme samples. Theoretically, we establish upper bounds on expected constraint violations during policy updates, guaranteeing strict constraint satisfaction at a zero-violation quantile level. Further, we demonstrate that EVO achieves a lower probability of constraint violations than expectation-based methods and exhibits lower variance than quantile regression methods. Extensive experiments show that EVO significantly reduces constraint violations during training while maintaining competitive policy performance compared to baselines.
Paper Structure (38 sections, 9 theorems, 77 equations, 12 figures, 3 tables, 1 algorithm)

This paper contains 38 sections, 9 theorems, 77 equations, 12 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Expectation Constraint.
  4. Probability Constraint.
  5. Preliminaries
  6. Constrained Reinforcement Learning
  7. Extreme Value Theory
  8. Methodology
  9. Extreme Quantile Constraint
  10. Extreme Prioritization Resampling
  11. Importance Resample for Extreme Samples
  12. Theoretical Analysis
  13. Experiment
  14. Environment.
  15. Experiment Setting.
  16. ...and 23 more sections

Key Result

Theorem 3.1

$X_1, \cdots, X_n$ denote a sequence of IID random variables with cumulative distribution function $F$, which approaches the Generalized Pareto distribution (GPD) asymptotically. Denote the conditional excess distribution as $F_t(x) = P(X-t \le x| X>t)$, then: where $t$ is a threshold and $H(x)$ denotes the GPD, following:

Figures (12)

  • Figure 1: (a) Probability density function of cumulative cost. Even when the expected constraint satisfies the threshold, there remains a high probability of constraint violations. (b) Fitting cumulative cost distribution with GPD and Gaussian, GPD captures tails more accurately.
  • Figure 2: Extreme quantile constraint $q_{\mu+\nu}$ and EVT-based constraint $q_\mu + q^H_{\frac{\nu}{1-\mu}}$. The peak set $Y_\mu$, containing samples exceeding the safety boundary $q_\mu$, is used to fit GPDs. The extreme set $Z_C$, containing samples exceeding the risk boundary $q_\mu + q^H_{\frac{\nu}{1-\mu}}$, is used to compute EVT-based constraints for reducing violations and to calculate extreme priorities for exploiting extreme samples.
  • Figure 3: Comparison of EVO to baselines on Safety Gym. The x-axis is the total number of training steps, the y-axis is the average return or constraint. The solid line is the mean and the shaded area is the standard deviation. The dashed line is the constraint threshold which is 25.
  • Figure 4: Comparison of EVO to baselines on Safety MuJoCo. The x-axis is the total number of training steps, the y-axis is the average return or constraint. The solid line is the mean and the shaded area is the standard deviation. The dashed line is the constraint threshold which is 25.
  • Figure 5: (a)(b) Ablation Study, including ablating EVT-based Constraint Objectives and ablating extreme prioritization. (c)(d) Cost Limit Sensitivity.
  • ...and 7 more figures

Theorems & Definitions (9)

  • Theorem 3.1
  • Theorem 4.1: Constraint violation upper bound
  • Theorem 4.2: Constraint violation probability
  • Theorem 4.3: Variance of EVO
  • Lemma 2.1
  • Theorem 2.2: Constraint violation upper bound
  • Theorem 2.3: Constraint violation probability
  • Lemma 2.4
  • Theorem 2.5: Variance of EVO