Provably Optimal Reinforcement Learning under Safety Filtering
Donggeon David Oh, Duy P. Nguyen, Haimin Hu, Jaime F. Fisac
TL;DR
The paper addresses the challenge of ensuring strict safety in reinforcement learning without sacrificing long-term performance. It introduces a safety-filtered RL framework built on a Safety-Critical MDP (SC-MDP) and a corresponding filtered MDP $\,\mathcal{M}_{φ}$ that uses a perfect safety filter $φ$, guaranteeing categorical safety via a maximal invariant safe set $Ω^*$. The main theoretical result establishes a complete safety–performance separation: any ε-optimal policy on $\,\mathcal{M}_{φ}$, when executed through $φ$, is an ε-optimal policy on the SC-MDP, and standard RL convergence carries over to the filtered setting. Empirically, the approach validated on Safety Gymnasium yields zero safety violations with a rollout-based filter and achieves equal or better asymptotic performance compared to unfiltered baselines, supporting its practical viability. The work thus provides a principled, implementable recipe: train with the most permissive safety filter and deploy with the same filter, ensuring strict safety and optimal long-run performance in tandem.
Abstract
Recent advances in reinforcement learning (RL) enable its use on increasingly complex tasks, but the lack of formal safety guarantees still limits its application in safety-critical settings. A common practical approach is to augment the RL policy with a safety filter that overrides unsafe actions to prevent failures during both training and deployment. However, safety filtering is often perceived as sacrificing performance and hindering the learning process. We show that this perceived safety-performance tradeoff is not inherent and prove, for the first time, that enforcing safety with a sufficiently permissive safety filter does not degrade asymptotic performance. We formalize RL safety with a safety-critical Markov decision process (SC-MDP), which requires categorical, rather than high-probability, avoidance of catastrophic failure states. Additionally, we define an associated filtered MDP in which all actions result in safe effects, thanks to a safety filter that is considered to be a part of the environment. Our main theorem establishes that (i) learning in the filtered MDP is safe categorically, (ii) standard RL convergence carries over to the filtered MDP, and (iii) any policy that is optimal in the filtered MDP-when executed through the same filter-achieves the same asymptotic return as the best safe policy in the SC-MDP, yielding a complete separation between safety enforcement and performance optimization. We validate the theory on Safety Gymnasium with representative tasks and constraints, observing zero violations during training and final performance matching or exceeding unfiltered baselines. Together, these results shed light on a long-standing question in safety-filtered learning and provide a simple, principled recipe for safe RL: train and deploy RL policies with the most permissive safety filter that is available.