Learning-Enhanced Safeguard Control for High-Relative-Degree Systems: Robust Optimization under Disturbances and Faults

TL;DR

The paper tackles safe reinforcement learning for nonlinear control systems with high-relative-degree safety constraints and unknown disturbances or actuator faults. It introduces high-order reciprocal control barrier functions (HO-RCBFs) to certify safety under high-rel-degree constraints and develops an adaptive safeguarding policy that balances safety with performance via gradient manipulation and a dynamic safeguarding gain. An online learning framework with disturbance observers and actor-critic networks is provided, including a formal safety and stability analysis that guarantees forward invariance of the safe set and convergence of the closed-loop system under appropriate conditions. Simulation studies on inverted pendulum and mobile robotic tasks demonstrate robust safety guarantees and improved performance when employing the HO-RCBF-based safe RL with adaptive safeguarding. The approach offers a practical, scalable path to achieving safe, data-driven control in safety-critical, high-dimensional systems facing disturbances and faults.

Abstract

Merely pursuing performance may adversely affect the safety, while a conservative policy for safe exploration will degrade the performance. How to balance the safety and performance in learning-based control problems is an interesting yet challenging issue. This paper aims to enhance system performance with safety guarantee in solving the reinforcement learning (RL)-based optimal control problems of nonlinear systems subject to high-relative-degree state constraints and unknown time-varying disturbance/actuator faults. First, to combine control barrier functions (CBFs) with RL, a new type of CBFs, termed high-order reciprocal control barrier function (HO-RCBF) is proposed to deal with high-relative-degree constraints during the learning process. Then, the concept of gradient similarity is proposed to quantify the relationship between the gradient of safety and the gradient of performance. Finally, gradient manipulation and adaptive mechanisms are introduced in the safe RL framework to enhance the performance with a safety guarantee. Two simulation examples illustrate that the proposed safe RL framework can address high-relative-degree constraint, enhance safety robustness and improve system performance.

Figures (6)

• Figure 1: Gradient manipulation on $\sqrt{R^{-1}}\tilde{\mathcal{L}}_g \mathcal{B}$ when (a): $\mu = 0$; and (b): $\mu=0.5$.
• Figure 2: Velocity and acceleration trajectory of the mobile robot under $K_s = 0.1$, $K_s = 50$ and $K_s = 500$. The maximum value of the velocity is $v_{max} = 15 m/s$.
• Figure 3: Trajectories of the angle and angular velocity under four algorithms, i.e., classical RL, RL with penalty term, RL with safety filter and our method. (a) without disturbance/actuator fault; (b) with disturbance/actuator fault
• Figure 4: Position and velocity trajectories of mobile robot under RL, RCBF-based RL max2023safe, HO-RCBF-based RL starting from different initial states.
• Figure 5: Position and velocity trajectories of the mobile robot under RL, HO-RCBF-based RL with a constant safeguarding gain, HO-RCBF-based RL with an adaptive safeguarding gain starting from different initial states.

Theorems & Definitions (9)

• proof
• proof
• proof
• proof
• proof
• proof
• proof
• proof
• proof