Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mirror Descent Policy Optimisation for Robust Constrained Markov Decision Processes

David M. Bossens, Atsushi Nitanda

TL;DR

<3-5 sentence high-level_summary> This paper tackles Safe Reinforcement Learning under epistemic uncertainty by formulating robust constrained MDPs (RCMDPs) and introducing Mirror Descent Policy Optimisation (MDPO) that jointly optimises the policy and an adversarial transition kernel via a robust Lagrangian. It develops Robust_Sample-based_PMD-PD with Transition_Mirror_Ascent (TMA) inner optimization, along with Approximate_TMA, achieving the state-of-the-art sample-based convergence rates: $\tilde{\mathcal{O}}(1/T^{1/3})$ for rectangular RCMDPs and $\tilde{\mathcal{O}}(1/T^{1/5})$ for non-rectangular cases, and provides a practical MDPO_Robust_Lagrangian_-based implementation. The work also extends to continuous spaces using a continuous pseudo-KL divergence of occupancy and demonstrates strong empirical gains across Cartpole and Inventory Management domains, highlighting improved robustness and constrained performance. Overall, the approach offers scalable, theoretically grounded, single-simulator RCMDP optimization with practical deep-RL integrations.

Abstract

Safety is an essential requirement for reinforcement learning systems. The newly emerging framework of robust constrained Markov decision processes allows learning policies that satisfy long-term constraints while providing guarantees under epistemic uncertainty. This paper presents mirror descent policy optimisation for robust constrained Markov decision processes, making use of policy gradient techniques to optimise both the policy (as a maximiser) and the transition kernel (as an adversarial minimiser) on the Lagrangian representing a constrained Markov decision process. Our proposed algorithm obtains an $\tilde{\mathcal{O}}\left(1/T^{1/3}\right)$ convergence rate in the sample-based robust constrained Markov decision process setting. The paper also contributes an algorithm for approximate gradient descent in the space of transition kernels, which is of independent interest for designing adversarial environments in general Markov decision processes. Experiments confirm the benefits of mirror descent policy optimisation in constrained and unconstrained optimisation, and significant improvements are observed in robustness tests when compared to baseline policy optimisation algorithms.

Mirror Descent Policy Optimisation for Robust Constrained Markov Decision Processes

TL;DR

<3-5 sentence high-level_summary> This paper tackles Safe Reinforcement Learning under epistemic uncertainty by formulating robust constrained MDPs (RCMDPs) and introducing Mirror Descent Policy Optimisation (MDPO) that jointly optimises the policy and an adversarial transition kernel via a robust Lagrangian. It develops Robust_Sample-based_PMD-PD with Transition_Mirror_Ascent (TMA) inner optimization, along with Approximate_TMA, achieving the state-of-the-art sample-based convergence rates: for rectangular RCMDPs and for non-rectangular cases, and provides a practical MDPO_Robust_Lagrangian_-based implementation. The work also extends to continuous spaces using a continuous pseudo-KL divergence of occupancy and demonstrates strong empirical gains across Cartpole and Inventory Management domains, highlighting improved robustness and constrained performance. Overall, the approach offers scalable, theoretically grounded, single-simulator RCMDP optimization with practical deep-RL integrations.

Abstract

Safety is an essential requirement for reinforcement learning systems. The newly emerging framework of robust constrained Markov decision processes allows learning policies that satisfy long-term constraints while providing guarantees under epistemic uncertainty. This paper presents mirror descent policy optimisation for robust constrained Markov decision processes, making use of policy gradient techniques to optimise both the policy (as a maximiser) and the transition kernel (as an adversarial minimiser) on the Lagrangian representing a constrained Markov decision process. Our proposed algorithm obtains an convergence rate in the sample-based robust constrained Markov decision process setting. The paper also contributes an algorithm for approximate gradient descent in the space of transition kernels, which is of independent interest for designing adversarial environments in general Markov decision processes. Experiments confirm the benefits of mirror descent policy optimisation in constrained and unconstrained optimisation, and significant improvements are observed in robustness tests when compared to baseline policy optimisation algorithms.

Paper Structure

This paper contains 58 sections, 33 theorems, 150 equations, 18 figures, 12 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Related work
  3. Robust constrained objectives
  4. Adversarial transition kernels
  5. Policy optimisation algorithms
  6. Preliminaries
  7. PMD-PD for CMDPs
  8. Robust Sample-based PMD-PD for RCMDPs
  9. General assumptions
  10. Global optimality of Lagrangian TMA
  11. Analysis of Approximate TMA
  12. Analysis of Robust Sample-based PMD-PD
  13. Extension to non-rectangular uncertainty sets
  14. KL-based Mirror Descent in continuous state-action spaces
  15. MDPO-Robust-Lagrangian: a practical implementation
  16. ...and 43 more sections

Key Result

Lemma 1

Let $\mathbf{V}_{\pi,p}(\rho; \lambda)$ be the Lagrangian for a policy $\pi \in \Pi$, a transition kernel $p \in \mathcal{P}$ parametrised by $\xi$, and dual variables $\lambda \in \mathbb{R}^m$. Then the transition kernel has the gradient

Figures (18)

  • Figure 1: Test performance of MDP and CMDP algorithms obtained by applying the learned determinstic policy from the Cartpole domain after 20,000 time steps of training. The line and shaded area represent the mean and standard error across the perturbations for the particular distortion level.
  • Figure 2: Test performance of RMDP and RCMDP algorithms obtained by applying the learned determinstic policy from the Cartpole domain after 20,000 time steps of training. The line and shaded area represent the mean and standard error across the perturbations for the particular distortion level.
  • Figure 3: Test performance of MDP and CMDP algorithms in the Inventory Management domain using the deterministic policy on the test distortions.
  • Figure 4: Test performance of RMDP and RCMDP algorithms in the Inventory Management domain using the deterministic policy on the test distortions.
  • Figure 5: Test performance of MDP and CMDP algorithms in the 3-D Inventory Management domain using the deterministic policy on the test distortions.
  • ...and 13 more figures

Theorems & Definitions (63)

  • Definition 1: Rectangularity
  • Lemma 1: Lagrangian transition gradient theorem
  • proof
  • Lemma 2: Regret of LTMA
  • proof
  • Definition 2: Action next-state value function (Definition 3.2 in Li2023)
  • Lemma 3: Partial derivative for approximate TMA
  • proof
  • Lemma 4: Ascent property of TMA (Lemma 5.4 of Wangd)
  • Lemma 5: Ascent property of Approximate TMA
  • ...and 53 more