IntersectionZoo: Eco-driving for Benchmarking Multi-Agent Contextual Reinforcement Learning

TL;DR

This work proposes IntersectionZoo, a comprehensive benchmark suite for multi-agent CRL through the real-world application of cooperative eco-driving in urban road networks, and benchmarks popular multi-agent RL and human-like driving algorithms and demonstrates that the popular multi-agent RL algorithms struggle to generalize in CRL settings.

Abstract

Despite the popularity of multi-agent reinforcement learning (RL) in simulated and two-player applications, its success in messy real-world applications has been limited. A key challenge lies in its generalizability across problem variations, a common necessity for many real-world problems. Contextual reinforcement learning (CRL) formalizes learning policies that generalize across problem variations. However, the lack of standardized benchmarks for multi-agent CRL has hindered progress in the field. Such benchmarks are desired to be based on real-world applications to naturally capture the many open challenges of real-world problems that affect generalization. To bridge this gap, we propose IntersectionZoo, a comprehensive benchmark suite for multi-agent CRL through the real-world application of cooperative eco-driving in urban road networks. The task of cooperative eco-driving is to control a fleet of vehicles to reduce fleet-level vehicular emissions. By grounding IntersectionZoo in a real-world application, we naturally capture real-world problem characteristics, such as partial observability and multiple competing objectives. IntersectionZoo is built on data-informed simulations of 16,334 signalized intersections derived from 10 major US cities, modeled in an open-source industry-grade microscopic traffic simulator. By modeling factors affecting vehicular exhaust emissions (e.g., temperature, road conditions, travel demand), IntersectionZoo provides one million data-driven traffic scenarios. Using these traffic scenarios, we benchmark popular multi-agent RL and human-like driving algorithms and demonstrate that the popular multi-agent RL algorithms struggle to generalize in CRL settings.

Figures (8)

• Figure 1: Cooperative eco-driving at signalized intersections where the controlled vehicles (CVs) are operated by an RL policy (or policies) to minimize the fleet-wise emissions that include both CVs and human-driven vehicles (HDVs). CVs implicitly control HDVs through car-following dynamics and form locally cooperative teams for better system control.
• Figure 2: A schematic overview of IntersectionZoo divided into three architectural layers.
• Figure 3: Left: The signalized intersection in the intersection of Bosworth Street and Diamond Street in Salt Lake City, Utah. Right: The reconstructed intersection in simulation.
• Figure 4: For each intersection, default nearby intersections are added for realistic vehicle arrival processes subjected to nearby traffic signals.
• Figure 5: Emission benefit histograms of Salt Lake City under different RL algorithms when trained and evaluated on Salt Lake City CMDP. Percentages are relative to the human-driving baseline. Large y-axis counts are truncated for clarity, with total counts indicated on the plot. The spikes at 0% are in part due to the aforementioned zeroing of emissions benefits for any scenarios where throughput is reduced. The total approach count is also given for reference in each plot.