Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enforcing Cooperative Safety for Reinforcement Learning-based Mixed-Autonomy Platoon Control

Jingyuan Zhou, Longhao Yan, Jinhao Liang, Kaidi Yang

TL;DR

Simulation results show that the proposed control strategy can effectively enhance the system-level safety through CAV cooperation of a mixed-autonomy platoon with a minimal impact on control performance.

Abstract

It is recognized that the control of mixed-autonomy platoons comprising connected and automated vehicles (CAVs) and human-driven vehicles (HDVs) can enhance traffic flow. Among existing methods, Multi-Agent Reinforcement Learning (MARL) appears to be a promising control strategy because it can manage complex scenarios in real time. However, current research on MARL-based mixed-autonomy platoon control suffers from several limitations. First, existing MARL approaches address safety by penalizing safety violations in the reward function, thus lacking theoretical safety guarantees due to the black-box nature of RL. Second, few studies have explored the cooperative safety of multi-CAV platoons, where CAVs can be coordinated to further enhance the system-level safety involving the safety of both CAVs and HDVs. Third, existing work tends to make an unrealistic assumption that the behavior of HDVs and CAVs is publicly known and rationale. To bridge the research gaps, we propose a safe MARL framework for mixed-autonomy platoons. Specifically, this framework (i) characterizes cooperative safety by designing a cooperative Control Barrier Function (CBF), enabling CAVs to collaboratively improve the safety of the entire platoon, (ii) provides a safety guarantee to the MARL-based controller by integrating the CBF-based safety constraints into MARL through a differentiable quadratic programming (QP) layer, and (iii) incorporates a conformal prediction module that enables each CAV to estimate the unknown behaviors of the surrounding vehicles with uncertainty qualification. Simulation results show that our proposed control strategy can effectively enhance the system-level safety through CAV cooperation of a mixed-autonomy platoon with a minimal impact on control performance.

Enforcing Cooperative Safety for Reinforcement Learning-based Mixed-Autonomy Platoon Control

TL;DR

Simulation results show that the proposed control strategy can effectively enhance the system-level safety through CAV cooperation of a mixed-autonomy platoon with a minimal impact on control performance.

Abstract

It is recognized that the control of mixed-autonomy platoons comprising connected and automated vehicles (CAVs) and human-driven vehicles (HDVs) can enhance traffic flow. Among existing methods, Multi-Agent Reinforcement Learning (MARL) appears to be a promising control strategy because it can manage complex scenarios in real time. However, current research on MARL-based mixed-autonomy platoon control suffers from several limitations. First, existing MARL approaches address safety by penalizing safety violations in the reward function, thus lacking theoretical safety guarantees due to the black-box nature of RL. Second, few studies have explored the cooperative safety of multi-CAV platoons, where CAVs can be coordinated to further enhance the system-level safety involving the safety of both CAVs and HDVs. Third, existing work tends to make an unrealistic assumption that the behavior of HDVs and CAVs is publicly known and rationale. To bridge the research gaps, we propose a safe MARL framework for mixed-autonomy platoons. Specifically, this framework (i) characterizes cooperative safety by designing a cooperative Control Barrier Function (CBF), enabling CAVs to collaboratively improve the safety of the entire platoon, (ii) provides a safety guarantee to the MARL-based controller by integrating the CBF-based safety constraints into MARL through a differentiable quadratic programming (QP) layer, and (iii) incorporates a conformal prediction module that enables each CAV to estimate the unknown behaviors of the surrounding vehicles with uncertainty qualification. Simulation results show that our proposed control strategy can effectively enhance the system-level safety through CAV cooperation of a mixed-autonomy platoon with a minimal impact on control performance.

Paper Structure

This paper contains 25 sections, 37 equations, 7 figures, 1 table.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Multi-Agent Reinforcement Learning (MARL)
  4. Control Barrier Functions
  5. Problem Statement
  6. Mixed-Autonomy Platoons Environment Modeling
  7. Safety Concerns Associated with Mixed-Autonomy Platoon
  8. Cooperative Safe MARL-Based Control Design
  9. Multi-Agent DRL-based Controller
  10. Cooperative Control Barrier Function
  11. Conformal Behavior Prediction Module
  12. Acceleration Rate Predictors
  13. Conformal Prediction Regions
  14. Robust Differentiable Quadratic Programming Layer
  15. Combining of Conformal Prediction and Cooperative CBF
  16. ...and 10 more sections

Figures (7)

  • Figure 1: Methodological framework for our proposed controller designed for multiple CAVs in a mixed-autonomy platoon, where blue vehicles represent CAVs, grey vehicles denote HDVs, and the black vehicle indicates the head vehicle.
  • Figure 2: Conformal prediction results for the predicted acceleration with conformal prediction bounds for HDV $1$.
  • Figure 3: Accumulated training rewards per episode.
  • Figure 4: Safety-guaranteed regions associated with two safety-critical scenarios. The dark blue region represents the safety region for the CAVs equipped with MARL controllers but without the safety layer (M2). The light blue region indicates the enhanced safety regions achieved through our proposed method (M5). The white region denotes areas that are considered unsafe.
  • Figure 5: Values of the CBF candidates in the two safety-critical scenarios.
  • ...and 2 more figures

Theorems & Definitions (2)

  • Definition 1: Control Barrier Function ames2014control
  • Definition 2: System-Level Safety