Dynamic Model Predictive Shielding for Provably Safe Reinforcement Learning
Arko Banerjee, Kia Rahmani, Joydeep Biswas, Isil Dillig
TL;DR
DMPS integrates a dynamic, planner-driven recovery mechanism with a neural policy to achieve provably safe reinforcement learning in high-dimensional control tasks. By using a local planner to identify recovery actions that maximize short-term progress while considering long-term value, DMPS reduces shield usage and improves final performance, with recovery regret decaying exponentially with planning horizon $n$. The approach yields safety during training and deployment, and empirical results across static and dynamic benchmarks show DMPS outperforming MPS and several SRL/PSRL baselines, particularly in dynamic environments. Key theoretical guarantees include RR = O($\gamma^n$) under appropriate planner properties, and a training loop that enables the policy to imitate safe planner actions while learning task-driven behavior. The work advances safe RL by fusing planning and learning, achieving practical safety with improved learning efficiency and performance in continuous domains.
Abstract
Among approaches for provably safe reinforcement learning, Model Predictive Shielding (MPS) has proven effective at complex tasks in continuous, high-dimensional state spaces, by leveraging a backup policy to ensure safety when the learned policy attempts to take risky actions. However, while MPS can ensure safety both during and after training, it often hinders task progress due to the conservative and task-oblivious nature of backup policies. This paper introduces Dynamic Model Predictive Shielding (DMPS), which optimizes reinforcement learning objectives while maintaining provable safety. DMPS employs a local planner to dynamically select safe recovery actions that maximize both short-term progress as well as long-term rewards. Crucially, the planner and the neural policy play a synergistic role in DMPS. When planning recovery actions for ensuring safety, the planner utilizes the neural policy to estimate long-term rewards, allowing it to observe beyond its short-term planning horizon. Conversely, the neural policy under training learns from the recovery plans proposed by the planner, converging to policies that are both high-performing and safe in practice. This approach guarantees safety during and after training, with bounded recovery regret that decreases exponentially with planning horizon depth. Experimental results demonstrate that DMPS converges to policies that rarely require shield interventions after training and achieve higher rewards compared to several state-of-the-art baselines.