Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Deep Multi-agent Reinforcement Learning for Highway On-Ramp Merging in Mixed Traffic

Dong Chen, Mohammad Hajidavalloo, Zhaojian Li, Kaian Chen, Yongqiang Wang, Longsheng Jiang, Yue Wang

TL;DR

The paper tackles traffic throughput and safety in highway on-ramp merging under mixed traffic by casting the problem as a decentralized MARL task. It introduces a scalable MA2C-based framework with action masking, local rewards, and curriculum learning, augmented by a priority-based safety supervisor that predicts HDV motions and enforces safe actions. The approach is validated in a gym-like simulator across varying densities and extended to multiple through-lanes, consistently outperforming MPC and other MARL baselines in collision reduction and throughput. The work contributes a practical, open-source platform and demonstrates that safety-guided exploration and local credit assignment substantially improve learning efficiency and traffic performance in mixed-traffic merging.

Abstract

On-ramp merging is a challenging task for autonomous vehicles (AVs), especially in mixed traffic where AVs coexist with human-driven vehicles (HDVs). In this paper, we formulate the mixed-traffic highway on-ramp merging problem as a multi-agent reinforcement learning (MARL) problem, where the AVs (on both merge lane and through lane) collaboratively learn a policy to adapt to HDVs to maximize the traffic throughput. We develop an efficient and scalable MARL framework that can be used in dynamic traffic where the communication topology could be time-varying. Parameter sharing and local rewards are exploited to foster inter-agent cooperation while achieving great scalability. An action masking scheme is employed to improve learning efficiency by filtering out invalid/unsafe actions at each step. In addition, a novel priority-based safety supervisor is developed to significantly reduce collision rate and greatly expedite the training process. A gym-like simulation environment is developed and open-sourced with three different levels of traffic densities. We exploit curriculum learning to efficiently learn harder tasks from trained models under simpler settings. Comprehensive experimental results show the proposed MARL framework consistently outperforms several state-of-the-art benchmarks.

Deep Multi-agent Reinforcement Learning for Highway On-Ramp Merging in Mixed Traffic

TL;DR

The paper tackles traffic throughput and safety in highway on-ramp merging under mixed traffic by casting the problem as a decentralized MARL task. It introduces a scalable MA2C-based framework with action masking, local rewards, and curriculum learning, augmented by a priority-based safety supervisor that predicts HDV motions and enforces safe actions. The approach is validated in a gym-like simulator across varying densities and extended to multiple through-lanes, consistently outperforming MPC and other MARL baselines in collision reduction and throughput. The work contributes a practical, open-source platform and demonstrates that safety-guided exploration and local credit assignment substantially improve learning efficiency and traffic performance in mixed-traffic merging.

Abstract

On-ramp merging is a challenging task for autonomous vehicles (AVs), especially in mixed traffic where AVs coexist with human-driven vehicles (HDVs). In this paper, we formulate the mixed-traffic highway on-ramp merging problem as a multi-agent reinforcement learning (MARL) problem, where the AVs (on both merge lane and through lane) collaboratively learn a policy to adapt to HDVs to maximize the traffic throughput. We develop an efficient and scalable MARL framework that can be used in dynamic traffic where the communication topology could be time-varying. Parameter sharing and local rewards are exploited to foster inter-agent cooperation while achieving great scalability. An action masking scheme is employed to improve learning efficiency by filtering out invalid/unsafe actions at each step. In addition, a novel priority-based safety supervisor is developed to significantly reduce collision rate and greatly expedite the training process. A gym-like simulation environment is developed and open-sourced with three different levels of traffic densities. We exploit curriculum learning to efficiently learn harder tasks from trained models under simpler settings. Comprehensive experimental results show the proposed MARL framework consistently outperforms several state-of-the-art benchmarks.

Paper Structure

This paper contains 19 sections, 12 equations, 17 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Preliminaries of Reinforcement Learning (RL)
  4. Multi-agent Reinforcement Learning (MARL)
  5. Ramp Merging as MARL
  6. Problem Formulation
  7. MA2C for CAVs
  8. DNN Settings
  9. Priority-based Safety Enhancement
  10. Priority Assignment
  11. Priority-based Safety Supervisor
  12. NUMERICAL EXPERIMENTS
  13. Reward Function Designs
  14. Curriculum Learning
  15. Performance of the Priority-based Safety Supervisor
  16. ...and 4 more sections

Figures (17)

  • Figure 1: Illustration of the considered on-ramp merging traffic scenario. CAVs (blue) and HDVs (green) coexist on both ramp and through lanes.
  • Figure 2: Schematics of system and simulation setup.
  • Figure 4: Architecture of the proposed network structure. The numbers in the parentheses denote the size of the layers. "w/o" and "w/" represent "without" and "with", respectively.
  • Figure 5: Illustration of trajectory conflict for $T_n=5$ steps.
  • Figure 6: Illustration of safety margin definitions. Top: safety margin if vehicle 1 turns left; and Bottom: safety margin when vehicle 1 keeps straight.
  • ...and 12 more figures

Theorems & Definitions (2)

  • Remark 1
  • Remark 2