Reinforcement Learning for Freeway Lane-Change Regulation via Connected Vehicles

TL;DR

This work tackles lane-change regulation in freeways under limited penetration of connected vehicles by introducing a centralized, macroscopic lane-grid MARL framework that broadcasts lane-change signals to CVs. It couples a multi-lane PDE-based traffic model with a SUMO-derived microscopic state, training grid-level agents via a Double DQN with shared parameters to regulate lane changes while preserving safety. The method demonstrates improved traffic efficiency and reduced lane-change activity across diverse demand scenarios, with a quantifiable trade-off in energy use and robustness to varying CV penetration. Its infrastructure-based regulation approach offers scalable applicability across mixed-autonomy traffic, highlighting practical benefits for real-world deployment while outlining avenues for improving compliance, generalization, and sim-to-real transfer.

Abstract

Lane change decision-making is a complex task due to intricate vehicle-vehicle and vehicle-infrastructure interactions. Existing algorithms for lane-change control often depend on vehicles with a certain level of autonomy (e.g., autonomous or connected autonomous vehicles). To address the challenges posed by low penetration rates of autonomous vehicles and the high costs of precise data collection, this study proposes a dynamic lane change regulation design based on multi-agent reinforcement learning (MARL) to enhance freeway traffic efficiency. The proposed framework leverages multi-lane macroscopic traffic models that describe spatial-temporal dynamics of the density and speed for each lane. Lateral traffic flow between adjacent lanes, resulting from aggregated lane-changing behaviors, is modeled as source terms exchanged between the partial differential equations (PDEs). We propose a lane change regulation strategy using MARL, where one agent is placed at each discretized lane grid. The state of each agent is represented by aggregated vehicle attributes within its grid, generated from the SUMO microscopic simulation environment. The agent's actions are lane-change regulations for vehicles in its grid. Specifically, lane-change regulation signals are computed at a centralized traffic management center and then broadcast to connected vehicles in the corresponding lane grids. Compared to vehicle-level maneuver control, this approach achieves a higher regulation rate by leveraging vehicle connectivity while introducing no critical safety concerns, and accommodating varying levels of connectivity and autonomy within the traffic system. The proposed model is simulated and evaluated in varied traffic scenarios and demand conditions. Experimental results demonstrate that the method improves overall traffic efficiency with minimal additional energy consumption while maintaining driving safety.

Figures (7)

• Figure 1: Model demonstration of lane change regulation signals broadcast to drivers via traffic management center.
• Figure 2: Hierarchical architecture of the proposed lane-change regulation model, integrating the microscopic traffic simulator SUMO for vehicle-level dynamics and macroscopic traffic PDEs for aggregated lane-change decisions.
• Figure 3: Demand settings with the fundamental diagram.
• Figure 4: Average agent reward per episode during the training phase under varying demand environments. From left to right, the columns represent low, high, and congested high demand conditions. The top row shows the total reward, while the second and third rows present the rewards for terms $r_1$ and $r_2$, respectively. The solid lines indicate a moving average with a window size of 60 episodes.
• Figure 5: Boxplots of performance uplift across episodes under varied demand conditions. Each environment is evaluated over 200 episodes. From left to right, the plots represent low, high, and congested high demand conditions. The colored boxes represent the interquartile range (IQR) containing 50% of the data near the median. Outliers are shown as black diamonds.