Multi-Constraint Safe Reinforcement Learning via Closed-form Solution for Log-Sum-Exp Approximation of Control Barrier Functions

TL;DR

This work tackles the challenge of provable safety in safe reinforcement learning under multiple constraints. It introduces a Log-Sum-Exp composite control barrier function (CBF) to fuse many safety constraints into a single, differentiable surrogate, enabling a closed-form solution for the associated CBF-based QP. The key contributions are (1) the composite CBF formulation with h(x) = -(1/κ) ln( sum_i e^{-κ h_i(x)} ), (2) a closed-form safe policy u_s(x) = ū(x) + max{0, η(x)} (L_g h(x))^T with η defined from L_f h, L_g h, and α(h), and (3) integration of this safety layer into an end-to-end RL framework (e.g., SAC) to achieve provable safety with reduced computational burden compared to differentiable QP layers. Experiments in a multi-obstacle reachability task show that the method maintains safety while offering substantial training-time speedups, demonstrating scalability to large constraint sets and practical applicability to real-time robotic control.

Abstract

The safety of training task policies and their subsequent application using reinforcement learning (RL) methods has become a focal point in the field of safe RL. A central challenge in this area remains the establishment of theoretical guarantees for safety during both the learning and deployment processes. Given the successful implementation of Control Barrier Function (CBF)-based safety strategies in a range of control-affine robotic systems, CBF-based safe RL demonstrates significant promise for practical applications in real-world scenarios. However, integrating these two approaches presents several challenges. First, embedding safety optimization within the RL training pipeline requires that the optimization outputs be differentiable with respect to the input parameters, a condition commonly referred to as differentiable optimization, which is non-trivial to solve. Second, the differentiable optimization framework confronts significant efficiency issues, especially when dealing with multi-constraint problems. To address these challenges, this paper presents a CBF-based safe RL architecture that effectively mitigates the issues outlined above. The proposed approach constructs a continuous AND logic approximation for the multiple constraints using a single composite CBF. By leveraging this approximation, a close-form solution of the quadratic programming is derived for the policy network in RL, thereby circumventing the need for differentiable optimization within the end-to-end safe RL pipeline. This strategy significantly reduces computational complexity because of the closed-form solution while maintaining safety guarantees. Simulation results demonstrate that, in comparison to existing approaches relying on differentiable optimization, the proposed method significantly reduces training computational costs while ensuring provable safety throughout the training process.

Figures (4)

• Figure 1: An illustration of an end-to-end training safe RL framework.
• Figure 2: An illustration of safe policy networks with different safety layers. Subfigure (a) demonstrates the proposed framework where $N$ constraints are composited to $h(x)$ using a continuous Log-Sum-Exp approximation. The safety layer is analytical based on the closed-form solution of the composite CBF-based optimization. Subfigure (b) demonstrates the existing framework with differentiable QP layer. The safety layer solves the forward CBF-based optimization and computes the gradient during backpropagation.
• Figure 3: Performance of safe RL training and testing with the proposed method. Subfigure (a) illustrates $\min h_i, i = 1, \dots, I$ and the composite $h$ for each episode during training. The composite $h(x)$ under approximates $\min h_i$ and maintains positive in each episode. Subfigure (b) illustrates the successful trajectories during testing. The blue area in Subfigure (b) contains three obstacles, each with a different safe radius size. The colored squares denote the initial positions. The red circle represents the target area, while the colored lines indicate the testing trajectories, each starting from different initial positions near the origin.
• Figure 4: Evolution of $h_1,h_2,h_3$ and the composite $h$ over 1000 episodes. The safe learning during training is guaranteed.