Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-Agent Reinforcement Learning-based Cooperative Autonomous Driving in Smart Intersections

Taoyuan Yu, Kui Wang, Zongdian Li, Tao Yu, Kei Sakaguchi

TL;DR

The paper addresses safety and efficiency at unsignalized intersections in mixed-traffic scenarios by leveraging an RSU-centered V2I-enabled cooperative framework. It introduces a two-stage hybrid reinforcement learning approach: offline pre-training with conservative Q-learning and behavior cloning, followed by online fine-tuning using MAPPO with self-attention and role-based policies. Experiments in CARLA demonstrate extremely low failure rates across up to three controlled vehicles and strong generalization to a real intersection map. The approach reduces onboard computation requirements for vehicles, enhances safety and throughput, and shows strong potential for practical RSU-assisted cooperative driving deployment.

Abstract

Unsignalized intersections pose significant safety and efficiency challenges due to complex traffic flows. This paper proposes a novel roadside unit (RSU)-centric cooperative driving system leveraging global perception and vehicle-to-infrastructure (V2I) communication. The core of the system is an RSU-based decision-making module using a two-stage hybrid reinforcement learning (RL) framework. At first, policies are pre-trained offline using conservative Q-learning (CQL) combined with behavior cloning (BC) on collected dataset. Subsequently, these policies are fine-tuned in the simulation using multi-agent proximal policy optimization (MAPPO), aligned with a self-attention mechanism to effectively solve inter-agent dependencies. RSUs perform real-time inference based on the trained models to realize vehicle control via V2I communications. Extensive experiments in CARLA environment demonstrate high effectiveness of the proposed system, by: \textit{(i)} achieving failure rates below 0.03\% in coordinating three connected and autonomous vehicles (CAVs) through complex intersection scenarios, significantly outperforming the traditional Autoware control method, and \textit{(ii)} exhibiting strong robustness across varying numbers of controlled agents and shows promising generalization capabilities on other maps.

Multi-Agent Reinforcement Learning-based Cooperative Autonomous Driving in Smart Intersections

TL;DR

The paper addresses safety and efficiency at unsignalized intersections in mixed-traffic scenarios by leveraging an RSU-centered V2I-enabled cooperative framework. It introduces a two-stage hybrid reinforcement learning approach: offline pre-training with conservative Q-learning and behavior cloning, followed by online fine-tuning using MAPPO with self-attention and role-based policies. Experiments in CARLA demonstrate extremely low failure rates across up to three controlled vehicles and strong generalization to a real intersection map. The approach reduces onboard computation requirements for vehicles, enhances safety and throughput, and shows strong potential for practical RSU-assisted cooperative driving deployment.

Abstract

Unsignalized intersections pose significant safety and efficiency challenges due to complex traffic flows. This paper proposes a novel roadside unit (RSU)-centric cooperative driving system leveraging global perception and vehicle-to-infrastructure (V2I) communication. The core of the system is an RSU-based decision-making module using a two-stage hybrid reinforcement learning (RL) framework. At first, policies are pre-trained offline using conservative Q-learning (CQL) combined with behavior cloning (BC) on collected dataset. Subsequently, these policies are fine-tuned in the simulation using multi-agent proximal policy optimization (MAPPO), aligned with a self-attention mechanism to effectively solve inter-agent dependencies. RSUs perform real-time inference based on the trained models to realize vehicle control via V2I communications. Extensive experiments in CARLA environment demonstrate high effectiveness of the proposed system, by: \textit{(i)} achieving failure rates below 0.03\% in coordinating three connected and autonomous vehicles (CAVs) through complex intersection scenarios, significantly outperforming the traditional Autoware control method, and \textit{(ii)} exhibiting strong robustness across varying numbers of controlled agents and shows promising generalization capabilities on other maps.
Paper Structure (14 sections, 17 equations, 6 figures, 1 table)

This paper contains 14 sections, 17 equations, 6 figures, 1 table.

Table of Contents

  1. Introduction
  2. RSU-CAVs Cooperative System
  3. Hybrid Reinforcement Learning Framework
  4. Observation Space
  5. Action Space
  6. Reward Function
  7. Offline Pre-training: Networks and Algorithm
  8. Online Fine-tuning: Networks and Algorithm
  9. Experiments and Analysis
  10. Baselines and Evaluation Metrics
  11. Offline Pre-training Results
  12. Online Training Results
  13. Final Model Performance Evaluation and Generalization Analysis
  14. Conclusion and Future Works

Figures (6)

  • Figure 1: High-level system design of the RSU-CAVs cooperative system
  • Figure 2: Offline-online hybrid RL algorithm framework design
  • Figure 3: Experimental scenario and generalization scenario settings (a) CARLA example map, (b) Real intersection map
  • Figure 4: Offline pre-training results
  • Figure 5: Comparison of online training w/ and w/o offline RL, (a) reward, (b) success rate.
  • ...and 1 more figures