Safety Filtering While Training: Improving the Performance and Sample Efficiency of Reinforcement Learning Agents

TL;DR

The paper addresses ensuring safety in reinforcement learning by integrating model predictive safety filters into the training loop. It introduces three training-time modifications—filtering training actions, penalizing corrections, and safely resetting episodes—applied with robust MPSFs, and validates them on a simulated safe-control environment and real Crazyflie 2.0 quadrotor experiments. Results show improved sample efficiency, reduced training-time constraint violations, and smoother, higher-performing certified behavior, with real-world experiments confirming the practicality of the approach. The work provides practical guidelines and a reusable framework for incorporating safety filters into RL training for safety-critical systems.

Abstract

Reinforcement learning (RL) controllers are flexible and performant but rarely guarantee safety. Safety filters impart hard safety guarantees to RL controllers while maintaining flexibility. However, safety filters can cause undesired behaviours due to the separation between the controller and the safety filter, often degrading performance and robustness. In this paper, we analyze several modifications to incorporating the safety filter in training RL controllers rather than solely applying it during evaluation. The modifications allow the RL controller to learn to account for the safety filter, improving performance. This paper presents a comprehensive analysis of training RL with safety filters, featuring simulated and real-world experiments with a Crazyflie 2.0 drone. We examine how various training modifications and hyperparameters impact performance, sample efficiency, safety, and chattering. Our findings serve as a guide for practitioners and researchers focused on safety filters and safe RL.

Figures (9)

• Figure 1: RL training with the safety filter modifications in red. An RL agent takes in the current state $\textbf{x}_k$ and outputs a (potentially unsafe) action $\textbf{u}_{\text{uncert}, k}$. This action is passed to the safety filter, which outputs a minimally corrected safe action $\textbf{u}_{\text{cert}, k}$. This safe action is applied to the environment, and the difference between the safe and unsafe action is used to penalize the reward received by the RL agent. When an episode is completed, the environment is reset to a safe state using the safety filter.
• Figure 2: Experimental setup on a Crazyflie 2.0 drone. A PPO controller trained without a safety filter or constraint violation penalization (blue) tries to track a reference trajectory (black), but unforeseen interactions with the safety filter cause poor tracking performance. However, when trained with a safety filter (green, with $\alpha=1$), the behaviour is smoother and more performant. The constraints are shown in red.
• Figure 3: The return for the ablation study of the training modifications.
• Figure 4: The rate of change of the inputs (see multi-step) for the ablation study of the training modifications.
• Figure 5: The return versus the number of training steps on the ablation study of the training modifications.

Theorems & Definitions (1)

• Definition 1: Robust pos. control inv. (RPCI) set mpc_textbook