Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Systematic Benchmarking of SUMO Against Data-Driven Traffic Simulators

Erdao Liang

TL;DR

This work benchmarks a widely used model-based traffic simulator (SUMO) against state-of-the-art data-driven counterparts using large real-world datasets (WOMD/WOSAC). It introduces Waymo2SUMO, an automated pipeline that converts WOMD scenarios into SUMO road networks, traffic signals, and multi-agent dynamics, enabling scalable, closed-loop evaluation. Across short (8s) and long (60s) horizons, SUMO achieves a realism meta metric of 0.653 on WOSAC with far fewer tunable parameters than learning-based models, and demonstrates superior long-horizon stability with low collision and off-road rates. The study highlights complementary strengths of model-based and data-driven approaches and advocates hybrid, high-fidelity, efficient simulation for autonomous driving validation and development.

Abstract

This paper presents a systematic benchmarking of the model-based microscopic traffic simulator SUMO against state-of-the-art data-driven traffic simulators using large-scale real-world datasets. Using the Waymo Open Motion Dataset (WOMD) and the Waymo Open Sim Agents Challenge (WOSAC), we evaluate SUMO under both short-horizon (8s) and long-horizon (60s) closed-loop simulation settings. To enable scalable evaluation, we develop Waymo2SUMO, an automated pipeline that converts WOMD scenarios into SUMO simulations. On the WOSAC benchmark, SUMO achieves a realism meta metric of 0.653 while requiring fewer than 100 tunable parameters. Extended rollouts show that SUMO maintains low collision and offroad rates and exhibits stronger long-horizon stability than representative data-driven simulators. These results highlight complementary strengths of model-based and data-driven approaches for autonomous driving simulation and benchmarking.

Systematic Benchmarking of SUMO Against Data-Driven Traffic Simulators

TL;DR

This work benchmarks a widely used model-based traffic simulator (SUMO) against state-of-the-art data-driven counterparts using large real-world datasets (WOMD/WOSAC). It introduces Waymo2SUMO, an automated pipeline that converts WOMD scenarios into SUMO road networks, traffic signals, and multi-agent dynamics, enabling scalable, closed-loop evaluation. Across short (8s) and long (60s) horizons, SUMO achieves a realism meta metric of 0.653 on WOSAC with far fewer tunable parameters than learning-based models, and demonstrates superior long-horizon stability with low collision and off-road rates. The study highlights complementary strengths of model-based and data-driven approaches and advocates hybrid, high-fidelity, efficient simulation for autonomous driving validation and development.

Abstract

This paper presents a systematic benchmarking of the model-based microscopic traffic simulator SUMO against state-of-the-art data-driven traffic simulators using large-scale real-world datasets. Using the Waymo Open Motion Dataset (WOMD) and the Waymo Open Sim Agents Challenge (WOSAC), we evaluate SUMO under both short-horizon (8s) and long-horizon (60s) closed-loop simulation settings. To enable scalable evaluation, we develop Waymo2SUMO, an automated pipeline that converts WOMD scenarios into SUMO simulations. On the WOSAC benchmark, SUMO achieves a realism meta metric of 0.653 while requiring fewer than 100 tunable parameters. Extended rollouts show that SUMO maintains low collision and offroad rates and exhibits stronger long-horizon stability than representative data-driven simulators. These results highlight complementary strengths of model-based and data-driven approaches for autonomous driving simulation and benchmarking.

Paper Structure

This paper contains 31 sections, 4 equations, 13 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Simulation platforms and scenario standards
  4. Data-driven traffic simulation
  5. Benchmarks, datasets, and toolchains for closed-loop evaluation
  6. Methodology
  7. Overall Framework
  8. Scenario Static Context Setup
  9. Road Network Construction
  10. Traffic Signal Estimation
  11. Scenario Dynamic Agents Setup
  12. Agent Parameters Setup
  13. Extra Control Logics
  14. Results
  15. Demonstration of Static Context Setup Results
  16. ...and 16 more sections

Figures (13)

  • Figure 1: The overall framework.
  • Figure 2: The road network construction process.
  • Figure 3: Definition of nodes, edges, and connections in SUMO. (a) The general structure around a node, (b) A node representing an intersection, (c) A node representing an area with changing number of lanes, (d) A node representing an area of diverging or merging roads.
  • Figure 4: Illustration of extra control logics applied to agents with special behaviors.
  • Figure 5: Demonstration of the static road network construction method in diverse real-world scenarios. The title for each pair of sub-figures is the corresponding scenario ID in the WOMD.
  • ...and 8 more figures