Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Policy Gradient in Robust MDPs with Global Convergence Guarantee

Qiuhao Wang, Chin Pang Ho, Marek Petrik

TL;DR

This work tackles robust MDPs by introducing the Double-Loop Robust Policy Gradient (DRPG), a generic policy-gradient scheme with a guaranteed global optimum for tabular $s$-rectangular RMDPs. DRPG employs two nested loops: an outer loop performing projected policy-gradient updates and an inner loop approximately solving the worst-case transition maximization over a compact ambiguity set, with a decreasing tolerance sequence to ensure stability. The analysis leverages the Moreau envelope to obtain a differentiable surrogate and proves a gradient-dominance-type condition that yields convergence to an $\epsilon$-global optimum in $\mathcal{O}(\epsilon^{-4})$ steps; a parametric transition representation enhances inner-loop scalability. Empirical results on Garnet benchmarks and an inventory-management example confirm DRPG’s global convergence and its robustness advantage over non-robust policy gradients, illustrating practical applicability to larger domains.

Abstract

Robust Markov decision processes (RMDPs) provide a promising framework for computing reliable policies in the face of model errors. Many successful reinforcement learning algorithms build on variations of policy-gradient methods, but adapting these methods to RMDPs has been challenging. As a result, the applicability of RMDPs to large, practical domains remains limited. This paper proposes a new Double-Loop Robust Policy Gradient (DRPG), the first generic policy gradient method for RMDPs. In contrast with prior robust policy gradient algorithms, DRPG monotonically reduces approximation errors to guarantee convergence to a globally optimal policy in tabular RMDPs. We introduce a novel parametric transition kernel and solve the inner loop robust policy via a gradient-based method. Finally, our numerical results demonstrate the utility of our new algorithm and confirm its global convergence properties.

Policy Gradient in Robust MDPs with Global Convergence Guarantee

TL;DR

This work tackles robust MDPs by introducing the Double-Loop Robust Policy Gradient (DRPG), a generic policy-gradient scheme with a guaranteed global optimum for tabular -rectangular RMDPs. DRPG employs two nested loops: an outer loop performing projected policy-gradient updates and an inner loop approximately solving the worst-case transition maximization over a compact ambiguity set, with a decreasing tolerance sequence to ensure stability. The analysis leverages the Moreau envelope to obtain a differentiable surrogate and proves a gradient-dominance-type condition that yields convergence to an -global optimum in steps; a parametric transition representation enhances inner-loop scalability. Empirical results on Garnet benchmarks and an inventory-management example confirm DRPG’s global convergence and its robustness advantage over non-robust policy gradients, illustrating practical applicability to larger domains.

Abstract

Robust Markov decision processes (RMDPs) provide a promising framework for computing reliable policies in the face of model errors. Many successful reinforcement learning algorithms build on variations of policy-gradient methods, but adapting these methods to RMDPs has been challenging. As a result, the applicability of RMDPs to large, practical domains remains limited. This paper proposes a new Double-Loop Robust Policy Gradient (DRPG), the first generic policy gradient method for RMDPs. In contrast with prior robust policy gradient algorithms, DRPG monotonically reduces approximation errors to guarantee convergence to a globally optimal policy in tabular RMDPs. We introduce a novel parametric transition kernel and solve the inner loop robust policy via a gradient-based method. Finally, our numerical results demonstrate the utility of our new algorithm and confirm its global convergence properties.
Paper Structure (22 sections, 17 theorems, 162 equations, 3 figures, 2 algorithms)

This paper contains 22 sections, 17 theorems, 162 equations, 3 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Notations and Settings
  3. Solving the Outer Loop
  4. Double-Loop Robust Policy Gradient Method (DRPG)
  5. Convergence Analysis
  6. Solving the Inner Loop
  7. Value-iteration Approach
  8. Gradient-based Approach
  9. Inner Loop Global Optimality
  10. Scalability of Parametric Transition
  11. Experiments
  12. Experimental Setup
  13. Results and Discussion
  14. Conclusion
  15. Examples of Common Ambiguity Sets
  16. ...and 7 more sections

Key Result

Lemma 3.1

The objective function $J_{\bm{\rho}}(\bm{\pi},\bm{p})$ in prob_RMDP is $L_{\bm{\pi}}$-Lipschitz and $\ell_{\bm{\pi}}$-smooth in $\bm{\pi}$ with Furthermore, the objective $\Phi(\bm{\pi})$ is $\ell_{\bm{\pi}}$-weakly convex and $L_{\bm{\pi}}$-Lipschitz.

Figures (3)

  • Figure 1: The error of value functions computed by DRPG for three Garnet problems with different sizes.
  • Figure 2: DRPG with parameterization v.s. Non-robust Policy Gradient on the Inventory Management Problem
  • Figure 3: The error of value function computed by non-parametric DRPG for two Garnet problems with $s$-rectangular ambiguity.

Theorems & Definitions (39)

  • Definition 2.2: Occupancy measure
  • Definition 2.3
  • Definition 2.4: Weak Convexity
  • Lemma 3.1
  • Theorem 3.2
  • Theorem 3.3: Global convergence for DRPG
  • Lemma 4.1: Differentiability
  • Lemma 4.2: Smoothness
  • Lemma 4.3: Gradient dominance
  • Theorem 4.4
  • ...and 29 more