Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Achieving Zero Constraint Violation for Constrained Reinforcement Learning via Conservative Natural Policy Gradient Primal-Dual Algorithm

Qinbo Bai, Amrit Singh Bedi, Vaneet Aggarwal

TL;DR

This work tackles constrained Markov decision processes in large state-action spaces by introducing the Conservative Natural Policy Gradient Primal-Dual Algorithm (C-NPG-PDA), which enforces zero constraint violations through a conservative feasibility bound $J_g^i(\pi) \ge \kappa$ while achieving strong convergence for the reward objective. The method leverages a natural policy gradient framework with a primal-dual update, enhanced by a conservative constraint to bound the dual variable and bridge the constrained and unconstrained problems via occupancy measures; a compatible-function SGD procedure realizes the NPG direction. Under standard smoothness and bias-assumptions, the authors establish global convergence of the conservative Lagrangian and derive explicit sample-trajectory complexities, showing a reduction from the prior $O(1/\epsilon^6)$ to $O(1/\epsilon^4)$ in key regimes. The proposed approach demonstrates zero-violation behavior in experiments and attains competitive objective performance, suggesting practical impact for safe and efficient CMDP learning in high-dimensional settings. The results contribute a principled way to attain zero constraint violations with natural-policy-gradient-style algorithms in infinite-horizon CMDPs, with clear pathways to extending to richer parametrizations and tighter biases.

Abstract

We consider the problem of constrained Markov decision process (CMDP) in continuous state-actions spaces where the goal is to maximize the expected cumulative reward subject to some constraints. We propose a novel Conservative Natural Policy Gradient Primal-Dual Algorithm (C-NPG-PD) to achieve zero constraint violation while achieving state of the art convergence results for the objective value function. For general policy parametrization, we prove convergence of value function to global optimal upto an approximation error due to restricted policy class. We even improve the sample complexity of existing constrained NPG-PD algorithm \cite{Ding2020} from $\mathcal{O}(1/ε^6)$ to $\mathcal{O}(1/ε^4)$. To the best of our knowledge, this is the first work to establish zero constraint violation with Natural policy gradient style algorithms for infinite horizon discounted CMDPs. We demonstrate the merits of proposed algorithm via experimental evaluations.

Achieving Zero Constraint Violation for Constrained Reinforcement Learning via Conservative Natural Policy Gradient Primal-Dual Algorithm

TL;DR

This work tackles constrained Markov decision processes in large state-action spaces by introducing the Conservative Natural Policy Gradient Primal-Dual Algorithm (C-NPG-PDA), which enforces zero constraint violations through a conservative feasibility bound while achieving strong convergence for the reward objective. The method leverages a natural policy gradient framework with a primal-dual update, enhanced by a conservative constraint to bound the dual variable and bridge the constrained and unconstrained problems via occupancy measures; a compatible-function SGD procedure realizes the NPG direction. Under standard smoothness and bias-assumptions, the authors establish global convergence of the conservative Lagrangian and derive explicit sample-trajectory complexities, showing a reduction from the prior to in key regimes. The proposed approach demonstrates zero-violation behavior in experiments and attains competitive objective performance, suggesting practical impact for safe and efficient CMDP learning in high-dimensional settings. The results contribute a principled way to attain zero constraint violations with natural-policy-gradient-style algorithms in infinite-horizon CMDPs, with clear pathways to extending to richer parametrizations and tighter biases.

Abstract

We consider the problem of constrained Markov decision process (CMDP) in continuous state-actions spaces where the goal is to maximize the expected cumulative reward subject to some constraints. We propose a novel Conservative Natural Policy Gradient Primal-Dual Algorithm (C-NPG-PD) to achieve zero constraint violation while achieving state of the art convergence results for the objective value function. For general policy parametrization, we prove convergence of value function to global optimal upto an approximation error due to restricted policy class. We even improve the sample complexity of existing constrained NPG-PD algorithm \cite{Ding2020} from to . To the best of our knowledge, this is the first work to establish zero constraint violation with Natural policy gradient style algorithms for infinite horizon discounted CMDPs. We demonstrate the merits of proposed algorithm via experimental evaluations.
Paper Structure (30 sections, 16 theorems, 135 equations, 2 figures, 1 table, 2 algorithms)

This paper contains 30 sections, 16 theorems, 135 equations, 2 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Formulation
  4. Proposed Approach
  5. Gradient of Value Function and Properties
  6. Natural Policy Gradient Primal-Dual Method with Zero Constraint Violation
  7. Convergence Rate Results
  8. Global Convergence For NPG-PD Method
  9. Simulation
  10. Conclusion
  11. Proof of Helpful Lemmas
  12. Computation of the Gradient for Value Function
  13. Proof of Lemma \ref{['lem_smooth']}
  14. Proof of Lemma \ref{['lem_grad_bound']}
  15. Proof of NPG update direction
  16. ...and 15 more sections

Key Result

Lemma 1

Under Assumption ass_score, both the objective function $\bm{J}_r^{\pi_\theta}$ and the constraint function $\bm{J}_{g^i}^{\pi_\theta}$ are $L_J$-smooth w.r.t. $\theta$. Formally, where $L_J=\frac{M}{(1-\gamma)^2}+\frac{2G^2}{(1-\gamma)^3}$

Figures (2)

  • Figure 1: Comparison of objective and constraint violation between $\kappa=0.5$ and $\kappa=0$. For the constraint violation figure, we use the log axis to make zero constraint violation more obvious.
  • Figure 2: Comparison of objective and constraint violation between the proposed algorithm $\kappa=1$ and NPD-PD Ding2020$\kappa=0$. For the constraint violation figure, we use the log axis to make zero constraint violation more obvious.

Theorems & Definitions (34)

  • Remark 1
  • Lemma 1
  • Lemma 2
  • Remark 2
  • Remark 3
  • Lemma 3
  • Lemma 4
  • Remark 4
  • Lemma 5
  • Remark 5
  • ...and 24 more