Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning General Parameterized Policies for Infinite Horizon Average Reward Constrained MDPs via Primal-Dual Policy Gradient Algorithm

Qinbo Bai, Washim Uddin Mondal, Vaneet Aggarwal

TL;DR

This work is the first to delve into the regret and constraint violation analysis of average reward CMDPs with a general policy parametrization, and proposes a primal dual-based policy gradient algorithm that adeptly manages the constraints while ensuring a low regret guarantee toward achieving a global optimal policy.

Abstract

This paper explores the realm of infinite horizon average reward Constrained Markov Decision Processes (CMDPs). To the best of our knowledge, this work is the first to delve into the regret and constraint violation analysis of average reward CMDPs with a general policy parametrization. To address this challenge, we propose a primal dual-based policy gradient algorithm that adeptly manages the constraints while ensuring a low regret guarantee toward achieving a global optimal policy. In particular, our proposed algorithm achieves $\tilde{\mathcal{O}}({T}^{4/5})$ objective regret and $\tilde{\mathcal{O}}({T}^{4/5})$ constraint violation bounds.

Learning General Parameterized Policies for Infinite Horizon Average Reward Constrained MDPs via Primal-Dual Policy Gradient Algorithm

TL;DR

This work is the first to delve into the regret and constraint violation analysis of average reward CMDPs with a general policy parametrization, and proposes a primal dual-based policy gradient algorithm that adeptly manages the constraints while ensuring a low regret guarantee toward achieving a global optimal policy.

Abstract

This paper explores the realm of infinite horizon average reward Constrained Markov Decision Processes (CMDPs). To the best of our knowledge, this work is the first to delve into the regret and constraint violation analysis of average reward CMDPs with a general policy parametrization. To address this challenge, we propose a primal dual-based policy gradient algorithm that adeptly manages the constraints while ensuring a low regret guarantee toward achieving a global optimal policy. In particular, our proposed algorithm achieves objective regret and constraint violation bounds.
Paper Structure (23 sections, 20 theorems, 123 equations, 1 table, 2 algorithms)

This paper contains 23 sections, 20 theorems, 123 equations, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Formulation
  3. Proposed Algorithm
  4. Global Convergence Analysis
  5. Regret and Violation Analysis
  6. Conclusions
  7. Acknowledgement
  8. Proofs for Lemmas in Section \ref{['sec_method']}
  9. Proof of Lemma \ref{['lemma_grad_compute']}
  10. Proof of Lemma \ref{['lemma_good_estimator']}
  11. Proofs for the Section of Global Convergence Analysis
  12. Proof of Lemma \ref{['lemma_grad_est_bias']}
  13. Proof of Lemma \ref{['lem_performance_diff']}
  14. Proof of Lemma \ref{['lem_framework']}
  15. Proof of Lemma \ref{['lemma:41ss']}
  16. ...and 8 more sections

Key Result

Lemma 1

The gradient of $J_{\mathrm{L}}(\cdot, \lambda)$ is computed as,

Theorems & Definitions (35)

  • Definition 1
  • Definition 2
  • Lemma 1
  • Lemma 2
  • Remark 1
  • Lemma 3
  • Remark 2
  • Remark 3
  • Lemma 4
  • Lemma 5
  • ...and 25 more