Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Distributional Soft Actor-Critic with Harmonic Gradient for Safe and Efficient Autonomous Driving in Multi-lane Scenarios

Feihong Zhang, Guojian Zhan, Bin Shuai, Tianyi Zhang, Jingliang Duan, Shengbo Eben Li

TL;DR

This paper tackles safety-constrained reinforcement learning for autonomous driving in multi-lane scenarios by introducing Harmonic Policy Iteration (HPI), which harmonizes two gradient signals for efficient driving and safety constraints. By integrating HPI with the Distributional Soft Actor-Critic (DSAC) backbone, the proposed DSAC-H algorithm updates the policy via a harmonic gradient that mitigates gradient conflicts, while learning separate reward and cost value distributions. Empirical results from extensive multi-lane simulations show that DSAC-H achieves faster convergence and near-zero safety violations compared to DSAC, with favorable statistics on tracking accuracy, stability, and collision-free performance. The work advances practical Safe RL for real-world autonomous driving, offering a principled gradient-level approach to balance efficiency and safety in complex traffic environments.

Abstract

Reinforcement learning (RL), known for its self-evolution capability, offers a promising approach to training high-level autonomous driving systems. However, handling constraints remains a significant challenge for existing RL algorithms, particularly in real-world applications. In this paper, we propose a new safety-oriented training technique called harmonic policy iteration (HPI). At each RL iteration, it first calculates two policy gradients associated with efficient driving and safety constraints, respectively. Then, a harmonic gradient is derived for policy updating, minimizing conflicts between the two gradients and consequently enabling a more balanced and stable training process. Furthermore, we adopt the state-of-the-art DSAC algorithm as the backbone and integrate it with our HPI to develop a new safe RL algorithm, DSAC-H. Extensive simulations in multi-lane scenarios demonstrate that DSAC-H achieves efficient driving performance with near-zero safety constraint violations.

Distributional Soft Actor-Critic with Harmonic Gradient for Safe and Efficient Autonomous Driving in Multi-lane Scenarios

TL;DR

This paper tackles safety-constrained reinforcement learning for autonomous driving in multi-lane scenarios by introducing Harmonic Policy Iteration (HPI), which harmonizes two gradient signals for efficient driving and safety constraints. By integrating HPI with the Distributional Soft Actor-Critic (DSAC) backbone, the proposed DSAC-H algorithm updates the policy via a harmonic gradient that mitigates gradient conflicts, while learning separate reward and cost value distributions. Empirical results from extensive multi-lane simulations show that DSAC-H achieves faster convergence and near-zero safety violations compared to DSAC, with favorable statistics on tracking accuracy, stability, and collision-free performance. The work advances practical Safe RL for real-world autonomous driving, offering a principled gradient-level approach to balance efficiency and safety in complex traffic environments.

Abstract

Reinforcement learning (RL), known for its self-evolution capability, offers a promising approach to training high-level autonomous driving systems. However, handling constraints remains a significant challenge for existing RL algorithms, particularly in real-world applications. In this paper, we propose a new safety-oriented training technique called harmonic policy iteration (HPI). At each RL iteration, it first calculates two policy gradients associated with efficient driving and safety constraints, respectively. Then, a harmonic gradient is derived for policy updating, minimizing conflicts between the two gradients and consequently enabling a more balanced and stable training process. Furthermore, we adopt the state-of-the-art DSAC algorithm as the backbone and integrate it with our HPI to develop a new safe RL algorithm, DSAC-H. Extensive simulations in multi-lane scenarios demonstrate that DSAC-H achieves efficient driving performance with near-zero safety constraint violations.
Paper Structure (13 sections, 12 equations, 7 figures, 2 tables, 1 algorithm)

This paper contains 13 sections, 12 equations, 7 figures, 2 tables, 1 algorithm.

Table of Contents

  1. INTRODUCTION
  2. Preliminary
  3. Principles of RL and Safe RL
  4. Distributional Soft Actor-Critic
  5. Method
  6. Harmonic Policy Iteration
  7. DSAC-H Algorithm
  8. Simulation Verification
  9. Simulation Environment
  10. Training Settings
  11. Training Curves and Visualization Demos
  12. Statistical Results Analysis
  13. Conclusion

Figures (7)

  • Figure 1: Harmonic Policy Iteration Framework.
  • Figure 2: Schematic of the multi-lane scenarios.
  • Figure 3: Training curves in the multi-lane scenarios.
  • Figure 4: Visualization of two typical cases in multi-lane scenarios. Case 1 (the first row) involves following and lane-changing behaviors. Case 2 (the second row) involves an overtaking behavior.
  • Figure 5: Histogram of position tracking errors.
  • ...and 2 more figures