Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Boosting CVaR Policy Optimization with Quantile Gradients

Yudong Luo, Erick Delage

TL;DR

This work tackles the sample inefficiency of CVaR policy gradient by augmenting the CVaR objective with an expected VaR term, enabling a dynamic-programming style use of all trajectories. A novel VaR Bellman operator is derived and adapted to the Markovian policy class, and a proximal VaR-PG component is integrated with CVaR-PG to form a unified CVaR-VaR algorithm. The approach discretizes quantile levels, uses a monotone quantile value function, and employs multi-step advantage estimation to learn efficiently. Empirical results across Maze, LunarLander, and InvertedPendulum show that CVaR-VaR achieves faster convergence and stronger risk-averse performance than strong baselines, validating the practical impact for risk-sensitive RL.

Abstract

Optimizing Conditional Value-at-risk (CVaR) using policy gradient (a.k.a CVaR-PG) faces significant challenges of sample inefficiency. This inefficiency stems from the fact that it focuses on tail-end performance and overlooks many sampled trajectories. We address this problem by augmenting CVaR with an expected quantile term. Quantile optimization admits a dynamic programming formulation that leverages all sampled data, thus improves sample efficiency. This does not alter the CVaR objective since CVaR corresponds to the expectation of quantile over the tail. Empirical results in domains with verifiable risk-averse behavior show that our algorithm within the Markovian policy class substantially improves upon CVaR-PG and consistently outperforms other existing methods.

Boosting CVaR Policy Optimization with Quantile Gradients

TL;DR

This work tackles the sample inefficiency of CVaR policy gradient by augmenting the CVaR objective with an expected VaR term, enabling a dynamic-programming style use of all trajectories. A novel VaR Bellman operator is derived and adapted to the Markovian policy class, and a proximal VaR-PG component is integrated with CVaR-PG to form a unified CVaR-VaR algorithm. The approach discretizes quantile levels, uses a monotone quantile value function, and employs multi-step advantage estimation to learn efficiently. Empirical results across Maze, LunarLander, and InvertedPendulum show that CVaR-VaR achieves faster convergence and stronger risk-averse performance than strong baselines, validating the practical impact for risk-sensitive RL.

Abstract

Optimizing Conditional Value-at-risk (CVaR) using policy gradient (a.k.a CVaR-PG) faces significant challenges of sample inefficiency. This inefficiency stems from the fact that it focuses on tail-end performance and overlooks many sampled trajectories. We address this problem by augmenting CVaR with an expected quantile term. Quantile optimization admits a dynamic programming formulation that leverages all sampled data, thus improves sample efficiency. This does not alter the CVaR objective since CVaR corresponds to the expectation of quantile over the tail. Empirical results in domains with verifiable risk-averse behavior show that our algorithm within the Markovian policy class substantially improves upon CVaR-PG and consistently outperforms other existing methods.
Paper Structure (29 sections, 9 theorems, 63 equations, 3 figures, 3 tables, 4 algorithms)

This paper contains 29 sections, 9 theorems, 63 equations, 3 figures, 3 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Quantile, VaR and CVaR
  4. CVaR Optimization in RL
  5. Limitations of CVaR-PG.
  6. VaR Optimization in RL
  7. Boosting CVaR Optimization with VaR
  8. The New VaR Bellman Operator
  9. Reducing to Markovian Policy Class
  10. Augmenting CVaR-PG with VaR-PG
  11. Parameterization of quantile function $v$
  12. Multi-step advantage estimation
  13. Experiments
  14. Maze
  15. LunarLander
  16. ...and 14 more sections

Key Result

Proposition 3.0

$\mathcal{T}_{\epsilon,\kappa}^*$ is a contraction mapping for $v$ with step size $\eta\in(0,\kappa]$.

Figures (3)

  • Figure 1: (a) Maze. Visiting red state will receive a random reward, with mean $-1$. (b) LunarLander. Landing on the right part of the ground will receive a random reward with mean 0. (c) Inverted Pendulum. Staying in the region where $x>0.01$ will receive a random reward with mean 0 per step.
  • Figure 2: (a) Expected return (b) Long (risk-averse) path rate in Maze. Curves are averaged over 10 seeds with shaded regions indicating standard errors.
  • Figure 3: Expected return, risk-averse rate, and CVaR 0.2 of return in LunarLander and Inverted Pendulum. Curves are averaged over 10 seeds with shaded regions indicating standard errors.

Theorems & Definitions (14)

  • Proposition 3.0
  • Proposition 3.0
  • Proposition 3.0
  • proof
  • Proposition 3.0
  • Proposition A.1
  • proof
  • Lemma A.2
  • Proposition A.2
  • proof
  • ...and 4 more