Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Survey of Safe Reinforcement Learning and Constrained MDPs: A Technical Survey on Single-Agent and Multi-Agent Safety

Ankita Kushwaha, Kiran Ravish, Preeti Lamba, Pawan Kumar

TL;DR

The paper addresses Safe Reinforcement Learning within the CMDP framework, formalizing safety as constraints $J_{c^{(i)}}(\\pi)\\le d_i$ alongside the primary return $J(\\pi)$. It surveys foundational CMDP theory, including $\mathcal{L}(\\pi,\\lambda)$ duality and linear-programming occupancy formulations, and surveys state-of-the-art single-agent methods (Lagrangian-based optimization, safety shields, and risk-sensitive approaches) and SafeMARL extensions (MACPO, scalable MAPPO-Lagrangian, shielding, and Stackelberg methods). It also discusses five open challenges—zero-violation safe exploration, partial observability, decentralized safety, non-stationarity, and competitive/multi-agent safety—and positions CMDP as a unifying language bridging machine learning, control theory, and formal methods. The findings highlight a mature set of SafeRL tools for practical deployment in safety-critical domains and identify SafeMARL as a vibrant frontier requiring advances in theory, scalability, and robustness for real-world multi-agent systems.

Abstract

Safe Reinforcement Learning (SafeRL) is the subfield of reinforcement learning that explicitly deals with safety constraints during the learning and deployment of agents. This survey provides a mathematically rigorous overview of SafeRL formulations based on Constrained Markov Decision Processes (CMDPs) and extensions to Multi-Agent Safe RL (SafeMARL). We review theoretical foundations of CMDPs, covering definitions, constrained optimization techniques, and fundamental theorems. We then summarize state-of-the-art algorithms in SafeRL for single agents, including policy gradient methods with safety guarantees and safe exploration strategies, as well as recent advances in SafeMARL for cooperative and competitive settings. Additionally, we propose five open research problems to advance the field, with three focusing on SafeMARL. Each problem is described with motivation, key challenges, and related prior work. This survey is intended as a technical guide for researchers interested in SafeRL and SafeMARL, highlighting key concepts, methods, and open future research directions.

A Survey of Safe Reinforcement Learning and Constrained MDPs: A Technical Survey on Single-Agent and Multi-Agent Safety

TL;DR

The paper addresses Safe Reinforcement Learning within the CMDP framework, formalizing safety as constraints alongside the primary return . It surveys foundational CMDP theory, including duality and linear-programming occupancy formulations, and surveys state-of-the-art single-agent methods (Lagrangian-based optimization, safety shields, and risk-sensitive approaches) and SafeMARL extensions (MACPO, scalable MAPPO-Lagrangian, shielding, and Stackelberg methods). It also discusses five open challenges—zero-violation safe exploration, partial observability, decentralized safety, non-stationarity, and competitive/multi-agent safety—and positions CMDP as a unifying language bridging machine learning, control theory, and formal methods. The findings highlight a mature set of SafeRL tools for practical deployment in safety-critical domains and identify SafeMARL as a vibrant frontier requiring advances in theory, scalability, and robustness for real-world multi-agent systems.

Abstract

Safe Reinforcement Learning (SafeRL) is the subfield of reinforcement learning that explicitly deals with safety constraints during the learning and deployment of agents. This survey provides a mathematically rigorous overview of SafeRL formulations based on Constrained Markov Decision Processes (CMDPs) and extensions to Multi-Agent Safe RL (SafeMARL). We review theoretical foundations of CMDPs, covering definitions, constrained optimization techniques, and fundamental theorems. We then summarize state-of-the-art algorithms in SafeRL for single agents, including policy gradient methods with safety guarantees and safe exploration strategies, as well as recent advances in SafeMARL for cooperative and competitive settings. Additionally, we propose five open research problems to advance the field, with three focusing on SafeMARL. Each problem is described with motivation, key challenges, and related prior work. This survey is intended as a technical guide for researchers interested in SafeRL and SafeMARL, highlighting key concepts, methods, and open future research directions.

Paper Structure

This paper contains 17 sections, 3 theorems, 18 equations, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Safe Reinforcement Learning and Constrained MDPs: Foundations
  4. Markov Decision Processes (MDPs)
  5. Constrained Markov Decision Processes (CMDPs)
  6. Lagrangian formulation:
  7. Linear programming solution:
  8. Constraint Types and Safety Specifications
  9. Theoretical Results
  10. Safe exploration and probably safe learning:
  11. State-of-the-Art Methods in SafeRL and SafeMARL
  12. Lagrangian-based Policy Optimization
  13. Safety Shields and Action Correction
  14. Risk-Sensitive and Distributional Methods
  15. Safe Multi-Agent Reinforcement Learning (SafeMARL)
  16. ...and 2 more sections

Key Result

Theorem 3.1

For a finite CMDP with bounded rewards and costs, there exists an optimal policy $(\pi^*, \lambda^*)$ that attains the maximum in eq:cmdp_objective (and corresponding optimal dual variables). Moreover, there exists an optimal policy that is stationary (time-independent) and can be chosen to be deter

Theorems & Definitions (4)

  • Definition 1.1
  • Theorem 3.1: Optimal Policy for CMDP Altman1999
  • Theorem 3.2: Lagrange Duality Altman1999
  • Proposition 3.3: Policy Gradient for Constrained Objectives