Safety Assessment in Reinforcement Learning via Model Predictive Control

TL;DR

This work addresses safety in model-free reinforcement learning where explicit safety constraints are hard to specify. It introduces reversibility as a safety proxy and RL-SAVMPC, which uses Model Predictive Path Integral (MPPI) planning to verify that a proposed action can be reversed to the current state, relying only on black-box dynamics. Empirical results across two environments show that RL-SAVMPC achieves zero constraint violations while maintaining training progress comparable to PPO baselines and, in some cases, outperforms shielding with full constraint knowledge. The approach enables safe RL in settings with unknown or latent safety constraints, broadening applicability to real-world control tasks without requiring explicit safety specification.

Abstract

Model-free reinforcement learning approaches are promising for control but typically lack formal safety guarantees. Existing methods to shield or otherwise provide these guarantees often rely on detailed knowledge of the safety specifications. Instead, this work's insight is that many difficult-to-specify safety issues are best characterized by invariance. Accordingly, we propose to leverage reversibility as a method for preventing these safety issues throughout the training process. Our method uses model-predictive path integral control to check the safety of an action proposed by a learned policy throughout training. A key advantage of this approach is that it only requires the ability to query the black-box dynamics, not explicit knowledge of the dynamics or safety constraints. Experimental results demonstrate that the proposed algorithm successfully aborts before all unsafe actions, while still achieving comparable training progress to a baseline PPO approach that is allowed to violate safety.

Figures (4)

• Figure 1: Illustration of RL-SAVMPC. The agent is attempting to overtake the vehicle in red. First, RL-SAVMPC projects an action forward. It then ensures there is a set of actions back to the original state before execution.
• Figure 2: Block diagram of RL-SAVMPC. At state $x$, the algorithm samples an action from the RL Policy $f_\theta$. It then propagates the action forward to get the induced state $x'$. The safety planner tries to find a set of actions $U_{safe}$ which return the agent to the previous state. If such a sequence exists, the original action is sent to the agent. If they do not exist, RL-SAVMPC resamples from $f_\theta$
• Figure 3: Reward results from both the Continuous Cartpole and Two Dimensional Navigation environments. These results are averaged over 10 seeds, and smoothed. The shaded regions represent one standard deviation. In the Continuous Cartpole environment, RL-SAVMPC closely approximates the algorithm with full constraint knowledge, and both produce similar results to baseline PPO. In the Two Dimensional Navigation environment, RL-SAVMPC outperforms PPO with full constraint knowledge, and matches the baseline PPO implementation.
• Figure 4: Constraint violations and aborts for each training environment. These results are averaged over 10 seeds, and smoothed. The shaded regions represent one standard deviation. \ref{['fig:cartpole-abort']} shows the results from the Cartpole Continuous environment. \ref{['fig:two-dim-nav-abort']} shows results from the Two Dimensional Navigation environment. In constraint violation figures, RL-SAVMPC is entirely overlapped by PPO with full constraint knowledge. These results show that neither shielding method produce any constraint violations. In the two dimensional navigation environment, RL-SAVMPC outperforms the second shielded algorithm

Theorems & Definitions (3)

• Definition IV.1: Trajectory
• Definition IV.2: Reversibility
• Definition IV.3: Safety