Safe But Not Sorry: Reducing Over-Conservatism in Safety Critics via Uncertainty-Aware Modulation

TL;DR

The paper tackles the safety–performance trade-off in reinforcement learning for real-world deployment, where existing safety critics either over-inflate costs or suppress exploration. It introduces Uncertain Safety Critic (USC), which modulates conservatism using parameter-space uncertainty and adds an uncertainty refinement step to sharpen cost predictions in uncertain regions. USC combines an uncertainty-weighted conservative objective with a refinement procedure powered by Gauss-Newton influence estimates and a log-sum-exp penalty to preserve gradient information in safe regions while remaining cautious where needed, and theory shows the LSE penalty becomes more conservative under uniform action sampling. Empirically, USC reduces safety violations by approximately $\approx 40\%$ and reduces gradient error by approximately $\approx 83\%$, while delivering competitive or higher rewards across Safety Gymnasium, Gymnasium Robotics, and MuJoCo tasks, and demonstrates scalable deployment with high success rates.

Abstract

Ensuring the safe exploration of reinforcement learning (RL) agents is critical for deployment in real-world systems. Yet existing approaches struggle to strike the right balance: methods that tightly enforce safety often cripple task performance, while those that prioritize reward leave safety constraints frequently violated, producing diffuse cost landscapes that flatten gradients and stall policy improvement. We introduce the Uncertain Safety Critic (USC), a novel approach that integrates uncertainty-aware modulation and refinement into critic training. By concentrating conservatism in uncertain and costly regions while preserving sharp gradients in safe areas, USC enables policies to achieve effective reward-safety trade-offs. Extensive experiments show that USC reduces safety violations by approximately 40% while maintaining competitive or higher rewards, and reduces the error between predicted and true cost gradients by approximately 83%, breaking the prevailing trade-off between safety and performance and paving the way for scalable safe RL.

Figures (10)

• Figure 1: A qualitative comparison of the predictive cost maps from the SC, CSC, and USC methods in the CarGoal2 environments against the ground truth cost map.
• Figure 2: High-level USC overview, showing its integration into the safe RL training loop. The safety critic receives transitions from the environment and, alongside the reward critic, guides policy updates. Within USC, uncertainty estimation is used to compute influence scores $\tilde{u}(s,a)$, which drive targeted overestimation in the conservative loss $\mathcal{L}_C$ to preserve sharp, informative gradients. In parallel, the most uncertain state–action pairs are identified, interpolated with confident neighbours to form synthetic costs $\tilde{c}^u$, and incorporated into a trust-region update $\mathcal{L}_{\text{refine}}$. The Actor and Reward Critic Networks are updated as standard. Together, these mechanisms reduce over-conservatism while ensuring that gradient structure remains accurate and actionable. A detailed USC description is provided in Section \ref{['sec:USC']}.
• Figure 3: Pareto plot of average episodic cost versus average episodic reward across evaluated benchmarks. Each point denotes the mean performance of a method, with shaded ellipses representing standard deviation.
• Figure 4: Reliability curve (left) showing prediction error vs predicted uncertainty, and risk–coverage curve (right) showing prediction error as a function of coverage ordered by uncertainty, where lower coverage corresponds to keeping only the most confident predictions.
• Figure 5: Different robots used within the tasks of the evaluated test-suites.

Theorems & Definitions (1)

• Theorem 1