Efficient Policy Optimization in Robust Constrained MDPs with Iteration Complexity Guarantees

TL;DR

The paper tackles robust constrained MDPs (RCMDPs) under model misspecification, where strong duality may fail and existing robust methods are computationally intensive. It introduces RNPG, a KL-regularized natural policy gradient approach that reformulates the RCMDP objective to avoid binary search and utilizes a robust policy evaluator to compute worst-case values. The authors prove an iteration complexity of $O(\epsilon^{-2})$ (modulo problem-dependent constants) to obtain an $\epsilon$-suboptimal and feasible policy, outperforming prior Epigraph-based methods. Empirically, RNPG achieves higher rewards with guaranteed feasibility and significantly faster wall-clock times (4x–6x faster) across multiple RCMDP benchmarks, supporting its practical impact for safe RL in uncertain environments.

Abstract

Constrained decision-making is essential for designing safe policies in real-world control systems, yet simulated environments often fail to capture real-world adversities. We consider the problem of learning a policy that will maximize the cumulative reward while satisfying a constraint, even when there is a mismatch between the real model and an accessible simulator/nominal model. In particular, we consider the robust constrained Markov decision problem (RCMDP) where an agent needs to maximize the reward and satisfy the constraint against the worst possible stochastic model under the uncertainty set centered around an unknown nominal model. Primal-dual methods, effective for standard constrained MDP (CMDP), are not applicable here because of the lack of the strong duality property. Further, one cannot apply the standard robust value-iteration based approach on the composite value function either as the worst case models may be different for the reward value function and the constraint value function. We propose a novel technique that effectively minimizes the constraint value function--to satisfy the constraints; on the other hand, when all the constraints are satisfied, it can simply maximize the robust reward value function. We prove that such an algorithm finds a policy with at most $ε$ sub-optimality and feasible policy after $O(ε^{-2})$ iterations. In contrast to the state-of-the-art method, we do not need to employ a binary search, thus, we reduce the computation time by at least 4x for smaller value of discount factor ($γ$) and by at least 6x for larger value of $γ$.

Figures (7)

• Figure 1: Comparison of RNPG, RPPG and EPIRC-PGS on Garnet(15,20) environment. Here, we want to maximize the objective (vf), and want the constraint (cf) to be above the baseline.
• Figure 2: Comparison of RPPG and EPIRC-PGS on CRS environment. Here we want to maximize the objective (vf) while constraint (cf) being below the threshold line.
• Figure 3: Comparison of RPPG and EPIRC-PGS on CRS environment
• Figure 4: Effect of $\lambda$ on RNPG for the CRS environment
• Figure 5: Comparison of RPPG and EPIRC-PGS on Garnet(15,20) environment $\lambda=30$

Theorems & Definitions (14)

• Proposition 1
• Lemma 3.1
• Theorem 4.1
• Lemma 4.2
• Theorem 6.1
• proof
• Lemma B.1: wang2024policy
• Lemma B.2: wang2024policy
• Lemma B.3
• proof