Safety through Permissibility: Shield Construction for Fast and Safe Reinforcement Learning

TL;DR

The paper tackles safety in reinforcement learning by integrating safety into permissibility-based action elimination, yielding a framework where unsafe and non-permissible actions are blocked without altering the learning objective or requiring pre-computed models. By defining unsafe actions as part of the non-permissible set, the approach constructs action-based shields (UNP) that preserve safety while accelerating convergence to optimal policies. Empirical results across CartPole, Lane Keeping, and FlappyBird show that UNP shields achieve faster, more reliable learning with maintained safety compared to baselines and previous shield methods, often reducing training time by significant margins. This work lowers human design burden for safety in RL and enhances practical applicability to real-world safety-critical tasks through a simpler, more general shield construction paradigm.

Abstract

Designing Reinforcement Learning (RL) solutions for real-life problems remains a significant challenge. A major area of concern is safety. "Shielding" is a popular technique to enforce safety in RL by turning user-defined safety specifications into safe agent behavior. However, these methods either suffer from extreme learning delays, demand extensive human effort in designing models and safe domains in the problem, or require pre-computation. In this paper, we propose a new permissibility-based framework to deal with safety and shield construction. Permissibility was originally designed for eliminating (non-permissible) actions that will not lead to an optimal solution to improve RL training efficiency. This paper shows that safety can be naturally incorporated into this framework, i.e. extending permissibility to include safety, and thereby we can achieve both safety and improved efficiency. Experimental evaluation using three standard RL applications shows the effectiveness of the approach.

Figures (3)

• Figure 1: Experimental results for the CartPole environment. The left plot shows agent performance over time; the right plots each correspond to an action and show a test trace of the best performing policy for each agent. Transitions in which unsafe actions were taken have larger dots while all other transitions have small dots. The dashed lines indicate where the boundaries are between $S_{\text{danger}}$ and all other states. Note that the y-axis of the right plots is in radians.
• Figure 2: Experimental results for the Lane Keeping environment. For format details, see Figure \ref{['fig:cp_plot']}
• Figure 3: Experimental results for the FlappyBird environment. In the right plots, the black dashed lines mark the y-axis positions of the lower pipe (at $0$) and upper pipe (at $100$). For further format details, see Figure \ref{['fig:cp_plot']}