Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sample-Efficient Constrained Reinforcement Learning with General Parameterization

Washim Uddin Mondal, Vaneet Aggarwal

TL;DR

The Primal-Dual Accelerated Natural Policy Gradient (PD-ANPG) algorithm is developed that ensures an $\epsilon$ global optimality gap and $\epsilon$ constraint violation and achieves the theoretical lower bound in $\epsilon^{-1}$.

Abstract

We consider a constrained Markov Decision Problem (CMDP) where the goal of an agent is to maximize the expected discounted sum of rewards over an infinite horizon while ensuring that the expected discounted sum of costs exceeds a certain threshold. Building on the idea of momentum-based acceleration, we develop the Primal-Dual Accelerated Natural Policy Gradient (PD-ANPG) algorithm that ensures an $ε$ global optimality gap and $ε$ constraint violation with $\tilde{\mathcal{O}}((1-γ)^{-7}ε^{-2})$ sample complexity for general parameterized policies where $γ$ denotes the discount factor. This improves the state-of-the-art sample complexity in general parameterized CMDPs by a factor of $\mathcal{O}((1-γ)^{-1}ε^{-2})$ and achieves the theoretical lower bound in $ε^{-1}$.

Sample-Efficient Constrained Reinforcement Learning with General Parameterization

TL;DR

The Primal-Dual Accelerated Natural Policy Gradient (PD-ANPG) algorithm is developed that ensures an global optimality gap and constraint violation and achieves the theoretical lower bound in .

Abstract

We consider a constrained Markov Decision Problem (CMDP) where the goal of an agent is to maximize the expected discounted sum of rewards over an infinite horizon while ensuring that the expected discounted sum of costs exceeds a certain threshold. Building on the idea of momentum-based acceleration, we develop the Primal-Dual Accelerated Natural Policy Gradient (PD-ANPG) algorithm that ensures an global optimality gap and constraint violation with sample complexity for general parameterized policies where denotes the discount factor. This improves the state-of-the-art sample complexity in general parameterized CMDPs by a factor of and achieves the theoretical lower bound in .
Paper Structure (21 sections, 11 theorems, 75 equations, 1 table, 1 algorithm)

This paper contains 21 sections, 11 theorems, 75 equations, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Challenges and Key Insights
  3. Related Works
  4. Formulation
  5. Algorithm
  6. Analysis
  7. Local-to-Global Convergence Lemma
  8. Local Convergence of the Natural Policy Gradient
  9. Global Convergence of the Lagrange
  10. Decoupling the Objective and the Constraint Violation Rates
  11. Conclusions and Limitations
  12. Helper Lemma
  13. Proof of Lemma \ref{['lemma_unbiased']}
  14. Proof of Lemma \ref{['lemma_lagrange_convergence']}
  15. Proof of Lemma \ref{['lemma_5']}
  16. ...and 6 more sections

Key Result

Lemma 1

Let $\hat{J}_{c, \rho}(\theta)$, $\hat{\nabla}_{\omega}L_{\nu^{\pi_\theta}_\rho}(\omega, \theta, \lambda)$ be the estimates produced by Algorithm algo_sampling for a predefined set of parameters $(\omega, \theta, \lambda)$. The following equations hold.

Theorems & Definitions (20)

  • Lemma 1
  • Remark 1
  • Lemma 2
  • proof
  • Remark 2
  • Lemma 3
  • Lemma 4
  • Lemma 5
  • Corollary 1
  • Theorem 1
  • ...and 10 more