Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Model-Based Safe Reinforcement Learning with Time-Varying State and Control Constraints: An Application to Intelligent Vehicles

Xinglong Zhang, Yaoqian Peng, Biao Luo, Wei Pan, Xin Xu, Haibin Xie

TL;DR

This work tackles safe reinforcement learning for nonlinear, discrete-time systems under time-varying state and control constraints by introducing a barrier force-based control policy (BCP) combined with a model-based, multi-step policy evaluation (MPE). By augmenting the cost with barrier terms and reformulating the problem in an extended state, the authors achieve an unconstrained-like optimization while preserving safety through repulsive barrier forces. The barrier-based actor-critic (BAC) implementation provides convergent online learning with theoretical guarantees of safety, stability, and robustness to disturbances, and it demonstrates strong sim-to-real performance on differential-drive and Ackermann-drive intelligent-vehicle platforms, outperforming several state-of-the-art safe RL methods and competing MPC variants. The results highlight the practical impact of a model-based, barrier-informed RL framework for real-world autonomous navigation tasks in dynamic environments, with clear pathways toward extensions to model-free settings. Overall, the paper advances safe RL by integrating continuous barrier-based safety with multi-step predictive evaluation and rigorous stability guarantees applicable to time-varying constraints in intelligent-vehicle control.

Abstract

Recently, safe reinforcement learning (RL) with the actor-critic structure for continuous control tasks has received increasing attention. It is still challenging to learn a near-optimal control policy with safety and convergence guarantees. Also, few works have addressed the safe RL algorithm design under time-varying safety constraints. This paper proposes a safe RL algorithm for optimal control of nonlinear systems with time-varying state and control constraints. In the proposed approach, we construct a novel barrier force-based control policy structure to guarantee control safety. A multi-step policy evaluation mechanism is proposed to predict the policy's safety risk under time-varying safety constraints and guide the policy to update safely. Theoretical results on stability and robustness are proven. Also, the convergence of the actor-critic implementation is analyzed. The performance of the proposed algorithm outperforms several state-of-the-art RL algorithms in the simulated Safety Gym environment. Furthermore, the approach is applied to the integrated path following and collision avoidance problem for two real-world intelligent vehicles. A differential-drive vehicle and an Ackermann-drive one are used to verify offline deployment and online learning performance, respectively. Our approach shows an impressive sim-to-real transfer capability and a satisfactory online control performance in the experiment.

Model-Based Safe Reinforcement Learning with Time-Varying State and Control Constraints: An Application to Intelligent Vehicles

TL;DR

This work tackles safe reinforcement learning for nonlinear, discrete-time systems under time-varying state and control constraints by introducing a barrier force-based control policy (BCP) combined with a model-based, multi-step policy evaluation (MPE). By augmenting the cost with barrier terms and reformulating the problem in an extended state, the authors achieve an unconstrained-like optimization while preserving safety through repulsive barrier forces. The barrier-based actor-critic (BAC) implementation provides convergent online learning with theoretical guarantees of safety, stability, and robustness to disturbances, and it demonstrates strong sim-to-real performance on differential-drive and Ackermann-drive intelligent-vehicle platforms, outperforming several state-of-the-art safe RL methods and competing MPC variants. The results highlight the practical impact of a model-based, barrier-informed RL framework for real-world autonomous navigation tasks in dynamic environments, with clear pathways toward extensions to model-free settings. Overall, the paper advances safe RL by integrating continuous barrier-based safety with multi-step predictive evaluation and rigorous stability guarantees applicable to time-varying constraints in intelligent-vehicle control.

Abstract

Recently, safe reinforcement learning (RL) with the actor-critic structure for continuous control tasks has received increasing attention. It is still challenging to learn a near-optimal control policy with safety and convergence guarantees. Also, few works have addressed the safe RL algorithm design under time-varying safety constraints. This paper proposes a safe RL algorithm for optimal control of nonlinear systems with time-varying state and control constraints. In the proposed approach, we construct a novel barrier force-based control policy structure to guarantee control safety. A multi-step policy evaluation mechanism is proposed to predict the policy's safety risk under time-varying safety constraints and guide the policy to update safely. Theoretical results on stability and robustness are proven. Also, the convergence of the actor-critic implementation is analyzed. The performance of the proposed algorithm outperforms several state-of-the-art RL algorithms in the simulated Safety Gym environment. Furthermore, the approach is applied to the integrated path following and collision avoidance problem for two real-world intelligent vehicles. A differential-drive vehicle and an Ackermann-drive one are used to verify offline deployment and online learning performance, respectively. Our approach shows an impressive sim-to-real transfer capability and a satisfactory online control performance in the experiment.
Paper Structure (22 sections, 7 theorems, 53 equations, 12 figures, 6 tables, 1 algorithm)

This paper contains 22 sections, 7 theorems, 53 equations, 12 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. System Model and Constraints
  4. Control Objective
  5. Preliminary Reinforcement Learning Solutions
  6. Definition on Barrier Functions
  7. Safe RL with BCP and MPE
  8. Design of Safe RL with BCP and MPE
  9. Barrier-based Actor-Critic Implementation
  10. Theoretical results
  11. Safety and Stability Guarantees in Nominal Scenario
  12. Safety and Robustness Guarantees in Disturbed Scenario
  13. Convergence Analysis of BAC Learning Algorithm
  14. Simulation and experimental results
  15. Application to a Differential-Drive Vehicle: Offline Learning Scenario
  16. ...and 7 more sections

Key Result

Lemma 1

Define a relaxed barrier function of ${\mathcal{B}}_k^c(z)$ as where the relaxing factor $\kappa_b>0$ is a small positive number, $\bar{\sigma}_k={\rm min}_{i\in\mathbb{N}_{1}^{p_z}}-G^{i}_{k}(z)$, the function $\gamma_{b}(z,\bar{\sigma}_k)$ is strictly monotone and differentiable on $(-\infty,\kappa_b)$, and $\triangledown_z^2\gamma_{b}(z,\bar{\sigma}_k)\leq \

Figures (12)

  • Figure 1: The schematic diagram of the barrier-based actor-critic learning algorithm.
  • Figure 2: (a) Simulation scenario in Safety Gym: the objective is to move the vehicle (red) to the green region while avoiding two static obstacles (grey), the moving soft object (purple) is not considered in the controller design; (b) Experimental platform of the differential-drive vehicle and testing scenario.
  • Figure 3: A: The HongQi EHS3 autonomous driving experimental platform. B: The upper panel presents the road map with road boundary constraints, where "S" stands for the starting point, "E" stands for the ending point, and the red line is the reference path for path following control. The lower panel presents the corresponding state errors Compared with offline learning case, a significantly improved performance can be achieved by online policy learning. C: The upper panel presents the road map with collision avoidance scenario, while the lower panel gives the numerical state errors.
  • Figure 4: The experimental results on path-following and collision avoidance by NMPC-c ($\mu_p=5\cdot 10^{-3}$): the shorter line with a gray shade represents the route of the moving vehicle, while the longer line represents the trajectory of the ego vehicle. When using dense reference points ($d_r=0.07$m), the ego vehicle was unable to pass, but was successful when using sparse reference points. In the latter scenario, the collision avoidance process caused the ego vehicle to experience a short transient period of rapid speed variation.
  • Figure 5: The experimental results on path-following and collision avoidance by NMPC-e ($\mu_p=5\cdot 10^{-4}$): the shorter line with a gray shade represents the route of the moving vehicle, while the longer line represents the trajectory of the ego vehicle. When using dense reference points ($d_r=0.07$m), the ego vehicle was unable to pass, but was successful when using sparse reference points. In the latter scenario, the collision avoidance process caused the ego vehicle to experience a short transient period of rapid speed variation.
  • ...and 7 more figures

Theorems & Definitions (18)

  • Definition 1: Local stabilizability zhang2021robust
  • Definition 2: Multi-step safe control
  • Definition 3: Barrier function wills2004barrier
  • Lemma 1: Relaxed barrier function wills2004barrier
  • Proposition 1: Unconstrained control problem equivalence
  • Remark 1
  • Remark 2
  • Remark 3
  • Remark 4
  • Theorem 1: Convergence
  • ...and 8 more