Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CBF-RL: Safety Filtering Reinforcement Learning in Training with Control Barrier Functions

Lizhi Yang, Blake Werner, Massimiliano de Sa, Aaron D. Ames

TL;DR

The paper tackles safety violations in reinforcement learning for humanoid robots by introducing CBF-RL, a dual approach that combines training-time safety filtering with barrier-inspired reward shaping. By proving a continuous-to-discrete relationship, it enables a closed-form CBF-QP solution to be used efficiently during training, allowing the policy to internalize safety and reduce reliance on runtime filters at deployment. Empirically, it demonstrates faster convergence, safer exploration, and robustness to uncertainty in both 2D navigation and high-dimensional humanoid locomotion tasks, including zero-shot sim-to-real transfer on a Unitree G1. The results suggest that policies trained with CBF-RL can operate safely in the real world without an online safety filter, with broad implications for safe deployment of complex robots in uncertain environments.

Abstract

Reinforcement learning (RL), while powerful and expressive, can often prioritize performance at the expense of safety. Yet safety violations can lead to catastrophic outcomes in real-world deployments. Control Barrier Functions (CBFs) offer a principled method to enforce dynamic safety -- traditionally deployed online via safety filters. While the result is safe behavior, the fact that the RL policy does not have knowledge of the CBF can lead to conservative behaviors. This paper proposes CBF-RL, a framework for generating safe behaviors with RL by enforcing CBFs in training. CBF-RL has two key attributes: (1) minimally modifying a nominal RL policy to encode safety constraints via a CBF term, (2) and safety filtering of the policy rollouts in training. Theoretically, we prove that continuous-time safety filters can be deployed via closed-form expressions on discrete-time roll-outs. Practically, we demonstrate that CBF-RL internalizes the safety constraints in the learned policy -- both enforcing safer actions and biasing towards safer rewards -- enabling safe deployment without the need for an online safety filter. We validate our framework through ablation studies on navigation tasks and on the Unitree G1 humanoid robot, where CBF-RL enables safer exploration, faster convergence, and robust performance under uncertainty, enabling the humanoid robot to avoid obstacles and climb stairs safely in real-world settings without a runtime safety filter.

CBF-RL: Safety Filtering Reinforcement Learning in Training with Control Barrier Functions

TL;DR

The paper tackles safety violations in reinforcement learning for humanoid robots by introducing CBF-RL, a dual approach that combines training-time safety filtering with barrier-inspired reward shaping. By proving a continuous-to-discrete relationship, it enables a closed-form CBF-QP solution to be used efficiently during training, allowing the policy to internalize safety and reduce reliance on runtime filters at deployment. Empirically, it demonstrates faster convergence, safer exploration, and robustness to uncertainty in both 2D navigation and high-dimensional humanoid locomotion tasks, including zero-shot sim-to-real transfer on a Unitree G1. The results suggest that policies trained with CBF-RL can operate safely in the real world without an online safety filter, with broad implications for safe deployment of complex robots in uncertain environments.

Abstract

Reinforcement learning (RL), while powerful and expressive, can often prioritize performance at the expense of safety. Yet safety violations can lead to catastrophic outcomes in real-world deployments. Control Barrier Functions (CBFs) offer a principled method to enforce dynamic safety -- traditionally deployed online via safety filters. While the result is safe behavior, the fact that the RL policy does not have knowledge of the CBF can lead to conservative behaviors. This paper proposes CBF-RL, a framework for generating safe behaviors with RL by enforcing CBFs in training. CBF-RL has two key attributes: (1) minimally modifying a nominal RL policy to encode safety constraints via a CBF term, (2) and safety filtering of the policy rollouts in training. Theoretically, we prove that continuous-time safety filters can be deployed via closed-form expressions on discrete-time roll-outs. Practically, we demonstrate that CBF-RL internalizes the safety constraints in the learned policy -- both enforcing safer actions and biasing towards safer rewards -- enabling safe deployment without the need for an online safety filter. We validate our framework through ablation studies on navigation tasks and on the Unitree G1 humanoid robot, where CBF-RL enables safer exploration, faster convergence, and robust performance under uncertainty, enabling the humanoid robot to avoid obstacles and climb stairs safely in real-world settings without a runtime safety filter.
Paper Structure (9 sections, 2 theorems, 19 equations, 7 figures, 3 tables, 1 algorithm)

This paper contains 9 sections, 2 theorems, 19 equations, 7 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Contributions
  3. Related Work
  4. Background
  5. Dual Approach to CBF-RL
  6. CBF-RL for Safe Humanoid Locomotion
  7. Task Definition
  8. Hardware Experiments
  9. Conclusion

Key Result

Lemma 1

Suppose $h: \mathbb{R}^n \to \mathbb{R}$ is a $C^1$ CBF for the continuous-time single integrator $\dot q = v$, where $q, v \in \mathbb{R}^n$. Let $k_s: \mathbb{R}^n \to \mathbb{R}^n$ be any safe, locally Lipschitz controller for the continuous-time integrator, satisfying for some $\alpha > 0$. Let $\mathbf{ f }_{\mathrm{\Delta t}}: \mathbb{R}^n \times \mathbb{R}^n \to \mathbb{R}^n, (\mathbf{ q

Figures (7)

  • Figure 1: A humanoid robot trained to climb stairs with the CBF-RL framework. Safety is injected into training by both filtering the policy-proposed actions and also provide safety rewards in addition to task and regularization rewards. During deployment the CBF policy retains safe behavior without a runtime filter.
  • Figure 2: CBF-RL framework: For one given task, the user defines the safety barrier function $h(\mathbf{q})$ and the accompanying $\nabla h(\mathbf{ q })$. During training, the RL policy proposes action $\mathbf{ v }^{\mathrm{policy}}$; the CBF safety filter then calculates the closed-form solution of the CBF-QP $\mathbf{ v }^{\mathrm{filtered}}$ and the safety reward $r_{\mathrm{cbf}}$ based on proposed action $\mathbf{ v }^{\mathrm{policy}}$ and agent configuration $\mathbf{ q }$. The RL agent executes $\mathbf{ v }^{\mathrm{filtered}}$ in the massively-parallel discretized environment, and the policy is updated with the combination of task, regularization, and safety rewards $r = r_\mathrm{nominal} + r_\mathrm{cbf}$. During deployment, the policy is able to output safe actions $\mathbf{v}^{\mathrm{policy}}_{\mathrm{cbf}}$ without needing an explicit runtime filter.
  • Figure 3: Training progress with the Dual, Reward Only, Filter Only and Nominal methods. Dual and Filter Only achieve faster convergence and avoid training-time safety violations.
  • Figure 4: Trajectory comparisons of one example simulation. The black dot is the start and the yellow circle is the goal. Success = reaching the goal; failure = collision with obstacles or wall.
  • Figure 5: Robot trajectory, $h$, and command vs. actual velocity visualization. The robot avoids obstacles approximated as cylinders without a runtime safety filter and climb up stairs. The velocity plots show the robot modulating its own velocities despite the command and the $h$ plot quantifies safety.
  • ...and 2 more figures

Theorems & Definitions (8)

  • Definition 1
  • Definition 2
  • Definition 3
  • Lemma 1
  • proof
  • Theorem 1: Continuous to Discrete Safety
  • Remark 1
  • proof