Deep QP Safety Filter: Model-free Learning for Reachability-based Safety Filter

TL;DR

This work tackles safety in black-box dynamical systems by learning a time-discounted Hamilton–Jacobi reachability safety filter directly from transition data. It trains neural nets to approximate the discounted safety value $V^λ(x)$ and its derivative using contraction-based Bellman operators, and enforces safety through a QP with an aggressiveness parameter $α$, including an LP fallback. In the exact setting, the critic converges to the viscosity solution of the HJ PDE, and empirically remains robust when the true value is non-smooth, extending to hybrid systems. The learned safety filter reduces pre-convergence failures in RL and transfers across tasks via a single scalar $α$, delivering a practical, scalable model-free safety layer for control. Overall, the method combines principled reachability with data-driven learning to provide safe, efficient, and adaptable control for black-box systems.

Abstract

We introduce Deep QP Safety Filter, a fully data-driven safety layer for black-box dynamical systems. Our method learns a Quadratic-Program (QP) safety filter without model knowledge by combining Hamilton-Jacobi (HJ) reachability with model-free learning. We construct contraction-based losses for both the safety value and its derivatives, and train two neural networks accordingly. In the exact setting, the learned critic converges to the viscosity solution (and its derivative), even for non-smooth values. Across diverse dynamical systems -- even including a hybrid system -- and multiple RL tasks, Deep QP Safety Filter substantially reduces pre-convergence failures while accelerating learning toward higher returns than strong baselines, offering a principled and practical route to safe, model-free control.

Figures (4)

• Figure 1: Overview of the Deep QP Safety Filter learned purely from transition data. The Safety Critic maps state and constraints to a safety value and derivatives, used by a QP solver together with a raw reference input, in order to produce a safe control input. The example sequence illustrates an unstable system with bang-bang ($\pm 1$) reference commands, where the filtered commands under our method maintain safety.
• Figure 2: Comparison with analytic solutions. For each plot, the horizontal and vertical axes represent the position and velocity of the Double Integrator. From top to bottom, the first column shows the analytic solutions $V(x)$, $\max_{u\in\mathcal{U}}\partial V(x,u)$, and $\frac{\partial V}{\partial x}(x)g(x)$, while the remaining columns show the learned counterparts $v^\lambda_\theta(x)$, $b^\lambda_\phi(x)$, and $a^\lambda_\phi(x)$ under different $\delta t$.
• Figure 3: Effects of $\alpha$ on the aggressiveness. In both cases, $u_\text{raw}$ is switched from -1 to 1, and the two images in each row are captured at identical time steps. Larger $\alpha$ produces more aggressive behavior with faster acceleration and deceleration.
• Figure 4: Comparison Results of RL tasks with the learned safety filter against other baselines. Inverted Double Pendulum environment with modified rewards (a): $|x|_\text{base}$ and (b): $|v|_\text{base}$. Hopper with default reward (c). All parameters of PPO used across baselines are identical.

Theorems & Definitions (11)

• definition 1: Safe state and safe set
• corollary 1
• corollary 2
• corollary 3
• proof
• lemma 1: PDE for the discounted safety value function
• theorem 1: Advantage in discounted safety value
• proof
• theorem 2: $e^{-\lambda \cdot \delta t}$-contraction on the discounted safety value
• theorem 3: $e^{-\lambda \cdot \delta t}$-contraction on the derivatives