Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Soft Robust MDPs and Risk-Sensitive MDPs: Equivalence, Policy Gradient, and Sample Complexity

Runyu Zhang, Yang Hu, Na Li

TL;DR

The paper shows that soft robust MDPs and risk-sensitive MDPs are equivalent under a convex-duality-based penalty, enabling a unified treatment. It derives a policy gradient theorem for soft RMDPs and proves gradient-dominance and global convergence for exact policy gradient under direct parameterization, highlighting favorable guarantees relative to Markov risk measures. For the KL-soft RMDP, equivalent to an entropy-risk MDP, it introduces the offline robust learning algorithm RFZI based on Z-functions and a projected Z-Bellman operator, and provides finite-sample guarantees on performance. This coupling yields practical, theoretically grounded tools for robust decision-making under transition-uncertainty, with clear guidance on parameter choices and sample complexity, while outlining limitations and avenues for extending to large or continuous spaces.

Abstract

Robust Markov Decision Processes (MDPs) and risk-sensitive MDPs are both powerful tools for making decisions in the presence of uncertainties. Previous efforts have aimed to establish their connections, revealing equivalences in specific formulations. This paper introduces a new formulation for risk-sensitive MDPs, which assesses risk in a slightly different manner compared to the classical Markov risk measure (Ruszczyński 2010), and establishes its equivalence with a class of soft robust MDP (RMDP) problems, including the standard RMDP as a special case. Leveraging this equivalence, we further derive the policy gradient theorem for both problems, proving gradient domination and global convergence of the exact policy gradient method under the tabular setting with direct parameterization. This forms a sharp contrast to the Markov risk measure, known to be potentially non-gradient-dominant (Huang et al. 2021). We also propose a sample-based offline learning algorithm, namely the robust fitted-Z iteration (RFZI), for a specific soft RMDP problem with a KL-divergence regularization term (or equivalently the risk-sensitive MDP with an entropy risk measure). We showcase its streamlined design and less stringent assumptions due to the equivalence and analyze its sample complexity

Soft Robust MDPs and Risk-Sensitive MDPs: Equivalence, Policy Gradient, and Sample Complexity

TL;DR

The paper shows that soft robust MDPs and risk-sensitive MDPs are equivalent under a convex-duality-based penalty, enabling a unified treatment. It derives a policy gradient theorem for soft RMDPs and proves gradient-dominance and global convergence for exact policy gradient under direct parameterization, highlighting favorable guarantees relative to Markov risk measures. For the KL-soft RMDP, equivalent to an entropy-risk MDP, it introduces the offline robust learning algorithm RFZI based on Z-functions and a projected Z-Bellman operator, and provides finite-sample guarantees on performance. This coupling yields practical, theoretically grounded tools for robust decision-making under transition-uncertainty, with clear guidance on parameter choices and sample complexity, while outlining limitations and avenues for extending to large or continuous spaces.

Abstract

Robust Markov Decision Processes (MDPs) and risk-sensitive MDPs are both powerful tools for making decisions in the presence of uncertainties. Previous efforts have aimed to establish their connections, revealing equivalences in specific formulations. This paper introduces a new formulation for risk-sensitive MDPs, which assesses risk in a slightly different manner compared to the classical Markov risk measure (Ruszczyński 2010), and establishes its equivalence with a class of soft robust MDP (RMDP) problems, including the standard RMDP as a special case. Leveraging this equivalence, we further derive the policy gradient theorem for both problems, proving gradient domination and global convergence of the exact policy gradient method under the tabular setting with direct parameterization. This forms a sharp contrast to the Markov risk measure, known to be potentially non-gradient-dominant (Huang et al. 2021). We also propose a sample-based offline learning algorithm, namely the robust fitted-Z iteration (RFZI), for a specific soft RMDP problem with a KL-divergence regularization term (or equivalently the risk-sensitive MDP with an entropy risk measure). We showcase its streamlined design and less stringent assumptions due to the equivalence and analyze its sample complexity
Paper Structure (46 sections, 21 theorems, 140 equations, 6 figures, 1 table, 2 algorithms)

This paper contains 46 sections, 21 theorems, 140 equations, 6 figures, 1 table, 2 algorithms.

Table of Contents

  1. Problem Settings and Preliminaries
  2. Markov Decision Processes (MDPs).
  3. Robust MDPs (RMDPs) and Soft Robust MDPs.
  4. Convex Risk Measures follmer2002convex.
  5. Risk-Sensitive MDPs.
  6. Equivalence of Soft RMDPs and Risk-Sensitive MDPs
  7. Policy Gradient for Soft RMDPs
  8. Offline Reinforcement Learning of the KL-soft RMDP
  9. Offline robust reinforcement learning.
  10. Robust Fitted-Z Iteration (RFZI)
  11. The Z-functions.
  12. Function approximation and projected Z-Bellman operator.
  13. Approximate the projected Z-Bellman operator with empirical loss minimization.
  14. Sample complexity
  15. Conclusions and Discussions
  16. ...and 31 more sections

Key Result

Theorem 1

The function $\sigma:\mathbb{R}^{|\mathcal{S}|}\to \mathbb{R}$ is a convex risk measure if and only if there exists a "penalty function" $D(\cdot): \Delta^{|\mathcal{S}|}\to \mathbb{R}$ such that Further, the penalty function $D$ can be chosen to satisfy the condition $D(\widehat{\mu})\ge -\sigma(0)$ for any $\widehat{\mu} \in \Delta^{|\mathcal{S}|}$ and it can be taken to be convex and lower-sem

Figures (6)

  • Figure 1: An exemplary 14-state environment with high-, medium-, and low-risk zones.
  • Figure 2: Optimality gap curves for the exact policy gradient algorithm in different settings.
  • Figure 3: Illustrated policies learned by the exact policy gradient algorithm in different settings.
  • Figure 4: Robustness values of the generated policies with respect to different $\delta$. ${}^{\dagger}$Risk-neutral policy refers to the optimal policy of the risk-neutral MDP. Robust baseline refers to the optimal policy of the RMDP with KL-rectangular ambiguity set $\mathcal{P}_{\delta}$.
  • Figure 5: Simulation results for practical RFZI in the 100-state environment. (Left: optimality gap; Middle: average test reward; Right: robustness value.)
  • ...and 1 more figures

Theorems & Definitions (50)

  • Remark 1: Soft Robust MDP.
  • Theorem 1: Dual Representation Theorem follmer2002convex
  • Example 1: Entropy risk measure follmer2002convex
  • Lemma 1
  • Remark 2
  • Theorem 2: Equivalence of Soft RMDPs and Risk-Sensitive MDPs
  • Remark 3
  • Theorem 3: Policy gradient theorem
  • Corollary 1: Policy gradient for direct parameterization
  • Lemma 2: Gradient domination under direct parameterization
  • ...and 40 more