Long and Short-Term Constraints Driven Safe Reinforcement Learning for Autonomous Driving

TL;DR

This work tackles safety during the exploration phase of reinforcement learning for autonomous driving by introducing Long and Short-Term Constraints (LSTC). The approach couples a long-term safety cost constraint $C_{\pi}(\theta) \le b$ with a short-term trajectory feasibility constraint $B_{\pi}(\tau^{n}) \le 0$, optimized via a dual-constraint Lagrangian within an actor-critic framework augmented by a validation network. Experiments on the MetaDrive simulator show that LSTC substantially improves safety (lower episode cost) and maintains strong learning performance, outperforming CMDP-based and SafeRL-kit baselines across diverse scenarios. The results demonstrate the practical potential of state-aware safety during training for safer, more reliable end-to-end autonomous driving systems.

Abstract

Reinforcement learning (RL) has been widely used in decision-making and control tasks, but the risk is very high for the agent in the training process due to the requirements of interaction with the environment, which seriously limits its industrial applications such as autonomous driving systems. Safe RL methods are developed to handle this issue by constraining the expected safety violation costs as a training objective, but the occurring probability of an unsafe state is still high, which is unacceptable in autonomous driving tasks. Moreover, these methods are difficult to achieve a balance between the cost and return expectations, which leads to learning performance degradation for the algorithms. In this paper, we propose a novel algorithm based on the long and short-term constraints (LSTC) for safe RL. The short-term constraint aims to enhance the short-term state safety that the vehicle explores, while the long-term constraint enhances the overall safety of the vehicle throughout the decision-making process, both of which are jointly used to enhance the vehicle safety in the training process. In addition, we develop a safe RL method with dual-constraint optimization based on the Lagrange multiplier to optimize the training process for end-to-end autonomous driving. Comprehensive experiments were conducted on the MetaDrive simulator. Experimental results demonstrate that the proposed method achieves higher safety in continuous state and action tasks, and exhibits higher exploration performance in long-distance decision-making tasks compared with state-of-the-art methods.

Figures (5)

• Figure 1: Long and short-term constraints for AD and the state space definitions. The reliable state space $\mathcal{S}_{f}$ indicates the state set where no danger occurs. The infeasible state space $\mathcal{S}_{inf}$ indicates the state set where a danger is imminent or already present. The short solid and long dotted lines denote the short and long trajectories of the vehicle, respectively. The blue dots represents the state in the short trajectories.
• Figure 2: Safe RL method with dual-constraint optimization. The actor-critic framework includes the cost value network and validation network. The cost value network takes the current state $s$ as the input and evaluates the expected cost. The validation network takes a state trajectory $\tau$ as the input and validates the short-term feasibility of decisions. When the constraints are not satisfied, the Lagrange multipliers are updated. The Lagrange multipliers act as a penalty in participating in policy updating, in order to learn a safe policy.
• Figure 3: MetaDrive Environment. (a) The vehicle is smoothly and safely navigating on the road with the real-time cost value of 0. (b) A collision with another vehicle results in a cost value of +1.
• Figure 4: Driving trajectories in different scenarios. Sparse points in the trajectories denote the vehicle at fast speeds, and vice versa.
• Figure 5: Training process curves. (a) Step-episode reward curves. (b) Step-success rate curves. (c) Step-episode cost curves. (d) Step-feasible state rate curves