Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ProSh: Probabilistic Shielding for Model-free Reinforcement Learning

Edwin Hamel-De le Court, Gaspard Ohlmann, Francesco Belardinelli

TL;DR

This work tackles safe reinforcement learning under cost constraints by introducing Probabilistic Shielding via Risk Augmentation (ProSh), a model-free approach that augments the CMDP with a running risk budget and shields the agent's action distributions using a learned cost critic. Theoretical foundations show that shielded policies in a risk-augmented MDP preserve budget coherence and can achieve constrained optimality in deterministic settings, with safety guarantees depending on the backup critic accuracy. The authors instantiate ProSh as a TD3-based algorithm (ProSh-TD3) featuring a differentiable shield layer, a backup critic, and a flipping-style reward actor that preserves optimality in the deterministic case, and demonstrate strong safety performance across Safe RL benchmarks. The results indicate ProSh can hard-shield learning to respect cost constraints during training while delivering competitive rewards, offering practical safety guarantees without requiring environment simulators or abstractions.

Abstract

Safety is a major concern in reinforcement learning (RL): we aim at developing RL systems that not only perform optimally, but are also safe to deploy by providing formal guarantees about their safety. To this end, we introduce Probabilistic Shielding via Risk Augmentation (ProSh), a model-free algorithm for safe reinforcement learning under cost constraints. ProSh augments the Constrained MDP state space with a risk budget and enforces safety by applying a shield to the agent's policy distribution using a learned cost critic. The shield ensures that all sampled actions remain safe in expectation. We also show that optimality is preserved when the environment is deterministic. Since ProSh is model-free, safety during training depends on the knowledge we have acquired about the environment. We provide a tight upper-bound on the cost in expectation, depending only on the backup-critic accuracy, that is always satisfied during training. Under mild, practically achievable assumptions, ProSh guarantees safety even at training time, as shown in the experiments.

ProSh: Probabilistic Shielding for Model-free Reinforcement Learning

TL;DR

This work tackles safe reinforcement learning under cost constraints by introducing Probabilistic Shielding via Risk Augmentation (ProSh), a model-free approach that augments the CMDP with a running risk budget and shields the agent's action distributions using a learned cost critic. Theoretical foundations show that shielded policies in a risk-augmented MDP preserve budget coherence and can achieve constrained optimality in deterministic settings, with safety guarantees depending on the backup critic accuracy. The authors instantiate ProSh as a TD3-based algorithm (ProSh-TD3) featuring a differentiable shield layer, a backup critic, and a flipping-style reward actor that preserves optimality in the deterministic case, and demonstrate strong safety performance across Safe RL benchmarks. The results indicate ProSh can hard-shield learning to respect cost constraints during training while delivering competitive rewards, offering practical safety guarantees without requiring environment simulators or abstractions.

Abstract

Safety is a major concern in reinforcement learning (RL): we aim at developing RL systems that not only perform optimally, but are also safe to deploy by providing formal guarantees about their safety. To this end, we introduce Probabilistic Shielding via Risk Augmentation (ProSh), a model-free algorithm for safe reinforcement learning under cost constraints. ProSh augments the Constrained MDP state space with a risk budget and enforces safety by applying a shield to the agent's policy distribution using a learned cost critic. The shield ensures that all sampled actions remain safe in expectation. We also show that optimality is preserved when the environment is deterministic. Since ProSh is model-free, safety during training depends on the knowledge we have acquired about the environment. We provide a tight upper-bound on the cost in expectation, depending only on the backup-critic accuracy, that is always satisfied during training. Under mild, practically achievable assumptions, ProSh guarantees safety even at training time, as shown in the experiments.
Paper Structure (35 sections, 7 theorems, 72 equations, 14 figures, 1 table, 2 algorithms)

This paper contains 35 sections, 7 theorems, 72 equations, 14 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work.
  3. Background
  4. Key Concepts
  5. Reinforcement Learning Problems
  6. Shielding and State-Augmentation challenges: a motivation example.
  7. ProSh: Theoretical Foundations
  8. Risk-Augmented CMDPs
  9. Shielded Policies, Safety and Optimality
  10. From constrained to unconstrained optimisation.
  11. Safe RL through Probabilistic Shielding
  12. Learning Algorithms
  13. ProSh-TD3
  14. Experimental Evaluation
  15. Experimental setup
  16. ...and 20 more sections

Key Result

Theorem 1

Let $\mathcal{M}$ be a CMDP with cost discount factor $\gamma_c$, and let $Q_b^{*}$ be its optimal state–action cost function. Assume that $Q_b$ is a $Q$-value, and let $\Delta_b = ||Q_b-Q_b^*||_{L^\infty(\mathcal{S}\mathcal{A})}$. For any policy $\bar{\pi}\in\overline\Pi_{\mathrm{val}}$ that is $Q_

Figures (14)

  • Figure 1: CMDP $\mathcal{M}_1$ with cost threshold $d = 0.5$ and $\gamma_r= \gamma_c= 1$.
  • Figure 2: Comparison with Off-Policy Algorithms on Half Cheetah. Cost threshold $d=10$.
  • Figure 3: Comparison with On-Policy Algorithms on Half Cheetah. Cost threshold $d=10$.
  • Figure 4: Comparison with Off-Policy Algorithms on Hopper. Cost threshold $d=10$.
  • Figure 5: Comparison with On-Policy Algorithms on Hopper. Cost threshold $d=10$.
  • ...and 9 more figures

Theorems & Definitions (15)

  • Definition 1: RL Problems
  • Definition 2: Risk-augmented CMDP
  • Definition 3: Valued policy
  • Definition 4: $Q_b$-Shielded policies
  • Definition 5: Shield-map
  • Theorem 1: Safety Bounds for $Q_b$-shields
  • Definition 6: Projection onto the base CMDP
  • Theorem 2: Preservation
  • Theorem 3: Optimality of the shielded policies
  • Theorem 4: Safety and near-Optimality
  • ...and 5 more