Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Regret Analysis of Unichain Average Reward Constrained MDPs with General Parameterization

Anirudh Satheesh, Vaneet Aggarwal

TL;DR

This work tackles infinite-horizon average-reward CMDPs under the unichain setting with general policy parameterizations. It introduces a Primal-Dual Natural Actor–Critic algorithm that leverages MLMC estimators and a logarithmic burn-in to handle transient dynamics without mixing-time Oracles, achieving a regret and constraint-violation rate of $\tilde{O}(\sqrt{T})$. Core contributions include the first MLMC estimators for unichain Markov chains, a burn-in analysis that reduces sample complexity, and a rigorous finite-time regret bound that accounts for approximation errors in both policy and critic. The results substantially broaden the applicability of order-optimal CMDP guarantees to systems with transient states and complex function approximation, enabling scalable, constraint-aware RL in more realistic environments.

Abstract

We study infinite-horizon average-reward constrained Markov decision processes (CMDPs) under the unichain assumption and general policy parameterizations. Existing regret analyses for constrained reinforcement learning largely rely on ergodicity or strong mixing-time assumptions, which fail to hold in the presence of transient states. We propose a primal--dual natural actor--critic algorithm that leverages multi-level Monte Carlo (MLMC) estimators and an explicit burn-in mechanism to handle unichain dynamics without requiring mixing-time oracles. Our analysis establishes finite-time regret and cumulative constraint violation bounds that scale as $\tilde{O}(\sqrt{T})$, up to approximation errors arising from policy and critic parameterization, thereby extending order-optimal guarantees to a significantly broader class of CMDPs.

Regret Analysis of Unichain Average Reward Constrained MDPs with General Parameterization

TL;DR

This work tackles infinite-horizon average-reward CMDPs under the unichain setting with general policy parameterizations. It introduces a Primal-Dual Natural Actor–Critic algorithm that leverages MLMC estimators and a logarithmic burn-in to handle transient dynamics without mixing-time Oracles, achieving a regret and constraint-violation rate of . Core contributions include the first MLMC estimators for unichain Markov chains, a burn-in analysis that reduces sample complexity, and a rigorous finite-time regret bound that accounts for approximation errors in both policy and critic. The results substantially broaden the applicability of order-optimal CMDP guarantees to systems with transient states and complex function approximation, enabling scalable, constraint-aware RL in more realistic environments.

Abstract

We study infinite-horizon average-reward constrained Markov decision processes (CMDPs) under the unichain assumption and general policy parameterizations. Existing regret analyses for constrained reinforcement learning largely rely on ergodicity or strong mixing-time assumptions, which fail to hold in the presence of transient states. We propose a primal--dual natural actor--critic algorithm that leverages multi-level Monte Carlo (MLMC) estimators and an explicit burn-in mechanism to handle unichain dynamics without requiring mixing-time oracles. Our analysis establishes finite-time regret and cumulative constraint violation bounds that scale as , up to approximation errors arising from policy and critic parameterization, thereby extending order-optimal guarantees to a significantly broader class of CMDPs.
Paper Structure (39 sections, 23 theorems, 147 equations, 1 table, 1 algorithm)

This paper contains 39 sections, 23 theorems, 147 equations, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Challenges and Contributions
  3. MLMC for Unichain Markov Chains.
  4. Logarithmic Burn-in.
  5. Formulation
  6. Algorithm
  7. Primal-Dual Framework
  8. Key Algorithmic Innovations
  9. Multi-Level Monte Carlo for Unichain MDPs
  10. Technical Novelty
  11. Handling Transient States via Explicit Burn-in
  12. Technical Novelty.
  13. Estimation Procedures
  14. Critic Estimation
  15. Natural Policy Gradient (NPG) Estimation
  16. ...and 24 more sections

Key Result

Lemma 2

Let $(Z_t)_{t \ge 0}$ be a unichain Markov chain with stationary distribution $d_Z$. Let $g : \mathcal{Z} \to \mathbb{R}^d$ satisfy Let $C_1 = C_{\mathrm{tar}}$ if $Z_0 \in \mathcal{S}_R^\theta$ and $C_1 = C$ otherwise. Then for all $N \geq 1$,

Theorems & Definitions (26)

  • Definition 1: Stationary Distribution
  • Lemma 2: MSE of Markovian sample average for Unichain MDPs
  • Lemma 3: MLMC Estimator Properties
  • Lemma 4: Regret decomposition under burn-in
  • Definition 5: Critic Approximation Errors
  • Definition 6: Policy Approximation Error
  • Theorem 7: Unichain Critic Convergence with MLMC
  • Theorem 8: NPG Convergence for Unichain CMDPs
  • Theorem 9: Main Regret Bounds
  • Theorem 10: Theorem 4, ganesh2025regret
  • ...and 16 more