Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Truly No-Regret Learning in Constrained MDPs

Adrian Müller, Pragnya Alatur, Volkan Cevher, Giorgia Ramponi, Niao He

TL;DR

This work tackles safe online reinforcement learning in unknown finite-horizon CMDPs by proving last-iterate convergence of a regularized primal-dual scheme across arbitrary numbers of constraints and by designing a model-based, optimistic online algorithm. It introduces a regularized Lagrangian with entropy and dual-norm terms, derives closed-form policy updates, and establishes last-iterate convergence to the regularized optimum; crucially, it translates these results into sublinear strong regret bounds in the online setting without relying on error cancellations. The online algorithm leverages an optimistic model with truncated policy evaluation to preserve safety during learning, yielding a strong regret of order $\tilde{O}(K^{0.93})$ with high probability. Simulations compare the approach to vanilla primal-dual methods, demonstrating that error cancellations are not necessary for practical safety and that sublinear strong regret is achievable in CMDPs.

Abstract

Constrained Markov decision processes (CMDPs) are a common way to model safety constraints in reinforcement learning. State-of-the-art methods for efficiently solving CMDPs are based on primal-dual algorithms. For these algorithms, all currently known regret bounds allow for error cancellations -- one can compensate for a constraint violation in one round with a strict constraint satisfaction in another. This makes the online learning process unsafe since it only guarantees safety for the final (mixture) policy but not during learning. As Efroni et al. (2020) pointed out, it is an open question whether primal-dual algorithms can provably achieve sublinear regret if we do not allow error cancellations. In this paper, we give the first affirmative answer. We first generalize a result on last-iterate convergence of regularized primal-dual schemes to CMDPs with multiple constraints. Building upon this insight, we propose a model-based primal-dual algorithm to learn in an unknown CMDP. We prove that our algorithm achieves sublinear regret without error cancellations.

Truly No-Regret Learning in Constrained MDPs

TL;DR

This work tackles safe online reinforcement learning in unknown finite-horizon CMDPs by proving last-iterate convergence of a regularized primal-dual scheme across arbitrary numbers of constraints and by designing a model-based, optimistic online algorithm. It introduces a regularized Lagrangian with entropy and dual-norm terms, derives closed-form policy updates, and establishes last-iterate convergence to the regularized optimum; crucially, it translates these results into sublinear strong regret bounds in the online setting without relying on error cancellations. The online algorithm leverages an optimistic model with truncated policy evaluation to preserve safety during learning, yielding a strong regret of order with high probability. Simulations compare the approach to vanilla primal-dual methods, demonstrating that error cancellations are not necessary for practical safety and that sublinear strong regret is achievable in CMDPs.

Abstract

Constrained Markov decision processes (CMDPs) are a common way to model safety constraints in reinforcement learning. State-of-the-art methods for efficiently solving CMDPs are based on primal-dual algorithms. For these algorithms, all currently known regret bounds allow for error cancellations -- one can compensate for a constraint violation in one round with a strict constraint satisfaction in another. This makes the online learning process unsafe since it only guarantees safety for the final (mixture) policy but not during learning. As Efroni et al. (2020) pointed out, it is an open question whether primal-dual algorithms can provably achieve sublinear regret if we do not allow error cancellations. In this paper, we give the first affirmative answer. We first generalize a result on last-iterate convergence of regularized primal-dual schemes to CMDPs with multiple constraints. Building upon this insight, we propose a model-based primal-dual algorithm to learn in an unknown CMDP. We prove that our algorithm achieves sublinear regret without error cancellations.
Paper Structure (41 sections, 42 theorems, 210 equations, 3 figures, 2 tables, 1 algorithm)

This paper contains 41 sections, 42 theorems, 210 equations, 3 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Formulation
  4. Primal-Dual Scheme
  5. Last-Iterate Convergence
  6. Online Setup
  7. Optimistic Model
  8. Regret Analysis
  9. Strong vs. Weak Regret and Safety at any Time
  10. Simulation
  11. Baselines and Environment
  12. Results
  13. Conclusion
  14. Summary of Notation
  15. Extended Related Work
  16. ...and 26 more sections

Key Result

Lemma 4.1

Let $\eta, \tau < 1$ and $\lambda_{max} \geq H\Xi^{-1}$. The iterates in eq:reg-primaleq:reg-dual satisfy where

Figures (3)

  • Figure 1: Constraint violation and objective suboptimality of the vanilla primal-dual algorithm efroni2020exploration and our regularized version (\ref{['algo:rpg-pd-finite-learn']}). We present the values of the individual policies in each episode while learning the CMDP.
  • Figure 2: Vanilla primal-dual algorithm efroni2020exploration and our regularized version (\ref{['algo:rpg-pd-finite-learn']}). \ref{['fig:strong']} shows the strong regret; \ref{['fig:weak']} shows the weak regret. The weak regret regarding the objective can be negative, illustrating that the iterates are superoptimal but unsafe on average. Y-axes differ across plots. All results are averaged over $n=5$ independent runs, with plotted confidence intervals.
  • Figure 3: Vanilla dual algorithm efroni2020exploration and our regularized version (\ref{['eq:reg-dual1', 'eq:reg-dual2']}). \ref{['fig:strong-dual']} shows the strong regret; \ref{['fig:weak-dual']} shows the weak regret. Y-axes differ across plots. All results are averaged over $n=5$ independent runs, with plotted confidence intervals.

Theorems & Definitions (80)

  • Definition 4.1: Last-iterate convergence
  • Lemma 4.1: Regularized convergence
  • Lemma 4.2: Error bounds
  • Theorem 4.1: Last-iterate convergence
  • Lemma 5.1: Regularized convergence
  • Theorem 5.1: Regret bound
  • Remark 5.1
  • Remark 5.2
  • Definition 3.1
  • Lemma 3.1: Strong duality CMDP paternain2019strongduality
  • ...and 70 more