Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A policy gradient approach for optimization of smooth risk measures

Nithia Vijayan, Prashanth L. A

TL;DR

This work proposes two template policy gradient algorithms that optimize a smooth risk measure in on-policy and off-policy RL settings, respectively, and derives non-asymptotic bounds that quantify the rate of convergence of the proposed algorithms to a stationary point of the smoothrisk measure.

Abstract

We propose policy gradient algorithms for solving a risk-sensitive reinforcement learning (RL) problem in on-policy as well as off-policy settings. We consider episodic Markov decision processes, and model the risk using the broad class of smooth risk measures of the cumulative discounted reward. We propose two template policy gradient algorithms that optimize a smooth risk measure in on-policy and off-policy RL settings, respectively. We derive non-asymptotic bounds that quantify the rate of convergence of our proposed algorithms to a stationary point of the smooth risk measure. As special cases, we establish that our algorithms apply to optimization of mean-variance and distortion risk measures, respectively.

A policy gradient approach for optimization of smooth risk measures

TL;DR

This work proposes two template policy gradient algorithms that optimize a smooth risk measure in on-policy and off-policy RL settings, respectively, and derives non-asymptotic bounds that quantify the rate of convergence of the proposed algorithms to a stationary point of the smoothrisk measure.

Abstract

We propose policy gradient algorithms for solving a risk-sensitive reinforcement learning (RL) problem in on-policy as well as off-policy settings. We consider episodic Markov decision processes, and model the risk using the broad class of smooth risk measures of the cumulative discounted reward. We propose two template policy gradient algorithms that optimize a smooth risk measure in on-policy and off-policy RL settings, respectively. We derive non-asymptotic bounds that quantify the rate of convergence of our proposed algorithms to a stationary point of the smooth risk measure. As special cases, we establish that our algorithms apply to optimization of mean-variance and distortion risk measures, respectively.
Paper Structure (31 sections, 26 theorems, 97 equations, 1 figure, 1 table, 1 algorithm)

This paper contains 31 sections, 26 theorems, 97 equations, 1 figure, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Policy gradient template
  4. Main results
  5. Bounds for OnP-SF/OffP-SF
  6. Applications
  7. Distortion risk measures (DRM)
  8. On-policy DRM estimation
  9. Off-policy DRM estimation
  10. Convergence analysis
  11. Mean-variance risk measure (MVRM)
  12. On-policy MVRM estimation
  13. Off-policy MVRM estimation
  14. Convergence analysis
  15. Conclusions
  16. ...and 16 more sections

Key Result

Proposition 1

(OnP-SF/OffP-SF) Assume as:mse-as:smooth. Let $\{\theta_i,i=0,\cdots,N-1\}$ be the policy parameters generated by OnP-SF/OffP-SF, and let $\theta_R$ be chosen uniformly at random from this set. Let $\rho^*=\max_{\theta\in\mathbb{R}^d}\rho(\theta)$. Then where $L_\rho,L_\rho'$, and $C_1$ are as in as:mse-as:smooth.

Figures (1)

  • Figure 1: Examples of distortion functions

Theorems & Definitions (54)

  • Definition 1
  • Definition 2: $\epsilon$-stationary point
  • Proposition 1
  • Theorem 1
  • Remark 1
  • Remark 2
  • Remark 3
  • Lemma 1
  • Lemma 2
  • Lemma 3
  • ...and 44 more