Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Risk-Constrained Reinforcement Learning with Percentile Risk Criteria

Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, Marco Pavone

TL;DR

The paper develops reinforcement learning algorithms for risk-constrained MDPs where risk is captured via CVaR or chance constraints. By formulating a Lagrangian and augmenting the MDP with an extra constraint-tracking state, it derives trajectory-based policy gradient and actor-critic methods with convergence guarantees to locally optimal policies. The methods are extended to both CVaR- and chance-constrained settings and demonstrated on an optimal stopping problem and a personalized ad-recommendation simulator, showing improved tail risk characteristics at acceptable cost to mean performance. The work provides rigorous convergence analyses across multiple time scales and offers practical algorithms for risk-aware decision-making in stochastic environments.

Abstract

In many sequential decision-making problems one is interested in minimizing an expected cumulative cost while taking into account \emph{risk}, i.e., increased awareness of events of small probability and high consequences. Accordingly, the objective of this paper is to present efficient reinforcement learning algorithms for risk-constrained Markov decision processes (MDPs), where risk is represented via a chance constraint or a constraint on the conditional value-at-risk (CVaR) of the cumulative cost. We collectively refer to such problems as percentile risk-constrained MDPs. Specifically, we first derive a formula for computing the gradient of the Lagrangian function for percentile risk-constrained MDPs. Then, we devise policy gradient and actor-critic algorithms that (1) estimate such gradient, (2) update the policy in the descent direction, and (3) update the Lagrange multiplier in the ascent direction. For these algorithms we prove convergence to locally optimal policies. Finally, we demonstrate the effectiveness of our algorithms in an optimal stopping problem and an online marketing application.

Risk-Constrained Reinforcement Learning with Percentile Risk Criteria

TL;DR

The paper develops reinforcement learning algorithms for risk-constrained MDPs where risk is captured via CVaR or chance constraints. By formulating a Lagrangian and augmenting the MDP with an extra constraint-tracking state, it derives trajectory-based policy gradient and actor-critic methods with convergence guarantees to locally optimal policies. The methods are extended to both CVaR- and chance-constrained settings and demonstrated on an optimal stopping problem and a personalized ad-recommendation simulator, showing improved tail risk characteristics at acceptable cost to mean performance. The work provides rigorous convergence analyses across multiple time scales and offers practical algorithms for risk-aware decision-making in stochastic environments.

Abstract

In many sequential decision-making problems one is interested in minimizing an expected cumulative cost while taking into account \emph{risk}, i.e., increased awareness of events of small probability and high consequences. Accordingly, the objective of this paper is to present efficient reinforcement learning algorithms for risk-constrained Markov decision processes (MDPs), where risk is represented via a chance constraint or a constraint on the conditional value-at-risk (CVaR) of the cumulative cost. We collectively refer to such problems as percentile risk-constrained MDPs. Specifically, we first derive a formula for computing the gradient of the Lagrangian function for percentile risk-constrained MDPs. Then, we devise policy gradient and actor-critic algorithms that (1) estimate such gradient, (2) update the policy in the descent direction, and (3) update the Lagrange multiplier in the ascent direction. For these algorithms we prove convergence to locally optimal policies. Finally, we demonstrate the effectiveness of our algorithms in an optimal stopping problem and an online marketing application.

Paper Structure

This paper contains 37 sections, 13 theorems, 180 equations, 3 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Notation
  4. Problem Statement
  5. Lagrangian Approach and Reformulation
  6. A Trajectory-based Policy Gradient Algorithm
  7. Actor-Critic Algorithms
  8. Gradient w.r.t. the Policy Parameters $\theta$
  9. Gradient w.r.t. the Lagrangian Parameter $\lambda$
  10. Sub-Gradient w.r.t. the VaR Parameter $\nu$
  11. Convergence of Actor-Critic Methods
  12. Extension to Chance-Constrained Optimization of MDPs
  13. Policy Gradient Method
  14. Actor-Critic Method
  15. Examples
  16. ...and 22 more sections

Key Result

Theorem 7

Under Assumptions ass:finite_time--ass:steps_pg, the sequence of policy updates in Algorithm alg_traj converges almost surely to a locally optimal policy $\theta^*$ for the CVaR-constrained optimization problemas $k$ goes to infinity.

Figures (3)

  • Figure 1: Cost distributions for the policies learned by the CVaR-constrained and risk-neutral policy gradient and actor-critic algorithms. The left figure corresponds to the PG methods and the right figure corresponds to the AC algorithms.
  • Figure 2: Cost distributions for the policies learned by the chance-constrained and risk-neutral policy gradient and actor-critic algorithms. The left figure corresponds to the PG methods and the right figure corresponds to the AC algorithms.
  • Figure 3: Reward distributions for the policies learned by the CVaR-constrained and risk-neutral policy gradient and actor-critic algorithms. The left figure corresponds to the PG methods and the right figure corresponds to the AC algorithms.

Theorems & Definitions (17)

  • Definition 1
  • Definition 5
  • Theorem 7
  • Theorem 10
  • Lemma 11
  • Theorem 12
  • Theorem 13
  • Theorem 14
  • Theorem 15
  • Theorem 16
  • ...and 7 more