Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Provably Efficient Sample Complexity for Robust CMDP

Sourav Ganguly, Arnob Ghosh

TL;DR

This work addresses learning policies for robust CMDPs where safety constraints must hold under worst-case dynamics. It introduces an augmented RCMDP by appending the remaining utility budget to the state, restoring Markovian optimality, and develops Robust Constrained Value Iteration (RCVI) that leverages a generative model and dual representations to handle multiple uncertainty metrics. The authors prove the first finite-sample guarantees for RCMDPs, showing near-optimal sample complexity of $\tilde{O}(|S||A|H^5/\epsilon^2)$ for TV, $\chi^2$, and KL divergences (with corresponding forms) and provide discretization-based extensions and LP-based policy updates. Empirical results on CRS and Garnet validate the method, demonstrating faster convergence and guaranteed feasibility compared to existing RCMDP approaches, highlighting the practical impact for safe RL under model mismatch.

Abstract

We study the problem of learning policies that maximize cumulative reward while satisfying safety constraints, even when the real environment differs from a simulator or nominal model. We focus on robust constrained Markov decision processes (RCMDPs), where the agent must maximize reward while ensuring cumulative utility exceeds a threshold under the worst-case dynamics within an uncertainty set. While recent works have established finite-time iteration complexity guarantees for RCMDPs using policy optimization, their sample complexity guarantees remain largely unexplored. In this paper, we first show that Markovian policies may fail to be optimal even under rectangular uncertainty sets unlike the {\em unconstrained} robust MDP. To address this, we introduce an augmented state space that incorporates the remaining utility budget into the state representation. Building on this formulation, we propose a novel Robust constrained Value iteration (RCVI) algorithm with a sample complexity of $\mathcal{\tilde{O}}(|S||A|H^5/ε^2)$ achieving at most $ε$ violation using a generative model where $|S|$ and $|A|$ denote the sizes of the state and action spaces, respectively, and $H$ is the episode length. To the best of our knowledge, this is the {\em first sample complexity guarantee} for RCMDP. Empirical results further validate the effectiveness of our approach.

Provably Efficient Sample Complexity for Robust CMDP

TL;DR

This work addresses learning policies for robust CMDPs where safety constraints must hold under worst-case dynamics. It introduces an augmented RCMDP by appending the remaining utility budget to the state, restoring Markovian optimality, and develops Robust Constrained Value Iteration (RCVI) that leverages a generative model and dual representations to handle multiple uncertainty metrics. The authors prove the first finite-sample guarantees for RCMDPs, showing near-optimal sample complexity of for TV, , and KL divergences (with corresponding forms) and provide discretization-based extensions and LP-based policy updates. Empirical results on CRS and Garnet validate the method, demonstrating faster convergence and guaranteed feasibility compared to existing RCMDP approaches, highlighting the practical impact for safe RL under model mismatch.

Abstract

We study the problem of learning policies that maximize cumulative reward while satisfying safety constraints, even when the real environment differs from a simulator or nominal model. We focus on robust constrained Markov decision processes (RCMDPs), where the agent must maximize reward while ensuring cumulative utility exceeds a threshold under the worst-case dynamics within an uncertainty set. While recent works have established finite-time iteration complexity guarantees for RCMDPs using policy optimization, their sample complexity guarantees remain largely unexplored. In this paper, we first show that Markovian policies may fail to be optimal even under rectangular uncertainty sets unlike the {\em unconstrained} robust MDP. To address this, we introduce an augmented state space that incorporates the remaining utility budget into the state representation. Building on this formulation, we propose a novel Robust constrained Value iteration (RCVI) algorithm with a sample complexity of achieving at most violation using a generative model where and denote the sizes of the state and action spaces, respectively, and is the episode length. To the best of our knowledge, this is the {\em first sample complexity guarantee} for RCMDP. Empirical results further validate the effectiveness of our approach.

Paper Structure

This paper contains 37 sections, 24 theorems, 111 equations, 9 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Other Related Works
  3. Problem Formulation
  4. Markovian Policy can be sub-optimal
  5. Augmented RCMDP
  6. Algorithm
  7. Main Results and Analysis
  8. Main Results
  9. Analysis
  10. Extension
  11. Experiments
  12. Cost-based Garnet:
  13. Conclusion
  14. Experiments
  15. Constrained River-swim
  16. ...and 22 more sections

Key Result

Lemma 1

Markovian policies can be sub-optimal for rectangular uncertainty sets.

Figures (9)

  • Figure 1: Transition diagram for the nominal model of CMDP.
  • Figure 2: Robust value function update at each iteration on CRS environment
  • Figure 3: Robust cost function update at each iteration on CRS environment
  • Figure 4: Robust value function update at each iteration on Garnet environment
  • Figure 5: Robust cost function update at each iteration on Garnet environment
  • ...and 4 more figures

Theorems & Definitions (46)

  • Definition 1
  • Example 1
  • Lemma 1
  • proof
  • Theorem 1
  • Definition 2
  • Theorem 2
  • Theorem 3
  • Theorem 4
  • Lemma 2: Policy dependence only via $\alpha$
  • ...and 36 more