Real-World Deployment and Assessment of a Multi-Agent Reinforcement Learning-Based Variable Speed Limit Control System

TL;DR

The paper tackles the challenge of validating a MARL-based variable speed limit system in a real-world highway setting. It presents a full deployment pipeline that trains policies in simulation, augments them with invalid-action masking and safety guards, and operates them through an AI-DSS integrated with the TMC for live field control on I-24 with 67 gantries. Key findings include high autonomous control feasibility (up to 98% of decisions executed without guards), improved proactive warning accuracy and responsiveness (14% higher accuracy and 75% faster response than a benchmark), and promising safety benefits (crash reductions of up to 26% for crashes and 50% for secondary crashes) over the deployed corridor. The work demonstrates the practical viability of scalable MARL for real-world traffic control, delivers open-source code and datasets, and provides a foundation for future enhancements in driver compliance and policy distillation for VSL systems.

Abstract

This article presents the first field deployment of a multi-agent reinforcement learning (MARL) based variable speed limit (VSL) control system on Interstate 24 (I-24) near Nashville, Tennessee. We design and demonstrate a full pipeline from training MARL agents in a traffic simulator to a field deployment on a 17-mile segment of I-24 encompassing 67 VSL controllers. The system was launched on March 8th, 2024, and has made approximately 35 million decisions on 28 million trips in six months of operation. We apply an invalid action masking mechanism and several safety guards to ensure real-world constraints. The MARL-based implementation operates up to 98% of the time, with the safety guards overriding the MARL decisions for the remaining time. We evaluate the performance of the MARL-based algorithm in comparison to a previously deployed non-RL VSL benchmark algorithm on I-24. Results show that the MARL-based VSL control system achieves a superior performance. The accuracy of correctly warning drivers about slowing traffic ahead is improved by 14% and the response delay to non-recurrent congestion is reduced by 75%. The preliminary data shows that the VSL control system has reduced the crash rate by 26% and the secondary crash rate by 50%. We open-sourced the deployed MARL-based VSL algorithm at https://github.com/Lab-Work/marl-vsl-controller.

Figures (8)

• Figure 1: Deployment pipeline of our MARL-based VSL control system: Step 1: We trained 8 agents in a traffic micro-simulation software TransModeler on a 7-mile stretch of I-24 and then tested it with 34 agents on a 17-mile stretch of westbound I-24 with various simulation parameters. Step 2: We extracted the optimal policy learned from simulation and applied invalid action masking and safety guards to satisfy real-world constraints. Step 3: We tested the behavior of the proposed MARL-based VSL control algorithm in an open-loop manner, with continuous streaming of I-24 sensor data feeding into Artificial-Intelligence Decision Support System (AI-DSS), the infrastructure software served for communication with Traffic Management Center (TMC). Based on the testing results, we go back to Step 2 to refine our algorithm until it presents satisfying behaviors. Step 4: We deployed the MARL-based VSL control algorithm in a closed-loop manner across 67 VSL controllers spanning a 17-mile segment of I-24 on March 8, 2024. The MARL-based VSL control system is continuously operating on I-24 today, affecting nearly 160,000 daily commuters.
• Figure 2: Map overview of the deployed MARL-based VSL control system on I-24, with the left direction heading towards downtown Nashville and the right towards Murfreesboro. The radar detection system (RDS) units are distributed along the corridor to provide real-time traffic state data for MARL-based control algorithm. The VSL controllers are changing speed limits every 30 seconds. The overlapping segment shaded in red (from mile marker 58.7 to mile marker 62.7) is covered by I-24 MOTION, an ultra-high-resolution traffic observation system used in this study for performance evaluation. Seven VSL controllers on the westbound direction are covered by I-24 MOTION, which is used for further validation.
• Figure 3: The deployed VSL control algorithm, centered around a MARL policy, considers all real-world constraints. IAM represents "Invalid Action Masking" and SM represents "Speed-Matching".
• Figure 4: Overview of the communication between AI-DSS, the TMC software SmartWay CS, and the TMC operator.
• Figure 5: The deployed MARL-based VSL control system on I-24 westbound: From a driver's perspective, this figure shows four consecutive gantries that the driver encounters when approaching a congestion tail. As drivers move forward, they encounter sequentially reduced speed limits of 60 (top left), 50 (top right), 40 (bottom left), and 30 (bottom right) mph on each gantry, alerting them to the upcoming slow-down traffic patterns.