ICSFuzz: Collision Detector Bug Discovery in Autonomous Driving Simulators

TL;DR

This paper tackles the reliability gap in autonomous driving simulators by addressing ignored collision scenarios (ICSs), i.e., false negatives where the simulator’s collision detector fails to report actual collisions. It introduces ICSFuzz, a black-box fuzzing framework that starts from determined collision scenarios and uses a directed, step-wise mutation guided by collision-contributing factors to efficiently discover ICSs, using an IoU-based oracle to identify undetected collisions. Empirical evaluation in Carla shows that ICSFuzz discovers 10–20× more ICSs with 20–70× speedups compared to DriveFuzz, uncovering ICSs across 10 types and obtaining developer-confirmed bugs including CVEs. The work demonstrates the practical impact of validating simulator reliability, offering a targeted methodology to identify and fix latent ICS bugs that could compromise real-world autonomous driving systems.

Abstract

With the increasing adoption of autonomous vehicles, ensuring the reliability of autonomous driving systems (ADSs) deployed on autonomous vehicles has become a significant concern. Driving simulators have emerged as crucial platforms for testing autonomous driving systems, offering realistic, dynamic, and configurable environments. However, existing simulation-based ADS testers have largely overlooked the reliability of the simulators, potentially leading to overlooked violation scenarios and subsequent safety security risks during real-world deployment. In our investigations, we identified that collision detectors in simulators could fail to detect and report collisions in certain collision scenarios, referred to as ignored collision scenarios. This paper aims to systematically discover ignored collision scenarios to improve the reliability of autonomous driving simulators. To this end, we present ICSFuzz, a black-box fuzzing approach to discover ignored collision scenarios efficiently. Drawing upon the fact that the ignored collision scenarios are a sub-type of collision scenarios, our approach starts with the determined collision scenarios. Following the guidance provided by empirically studied factors contributing to collisions, we selectively mutate arbitrary collision scenarios in a step-wise manner toward the ignored collision scenarios and effectively discover them. We compare ICSFuzz with DriveFuzz, a state-of-the-art simulation-based ADS testing method, by replacing its oracle with our ignored-collision-aware oracle. The evaluation demonstrates that ICSFuzz outperforms DriveFuzz by finding 10-20x more ignored collision scenarios with a 20-70x speedup. All the discovered ignored collisions have been confirmed by developers with one CVE ID assigned.

Figures (8)

• Figure 1: Collision scenarios in Carla, a driving simulator.
• Figure 2: ADS Scenario Space. (A) The scenario space from the existing ADS testers' perspective, with only collision and non-collision scenarios. (B) The actual scenario space, where the actual collision scenario space contains a ignored scenario space overlooked by ADS testers. a: Existing ADS collision testing methods. b: ICSFuzz.
• Figure 3: Structure of AD simulator.
• Figure 4: Overview of ICSFuzz.
• Figure 5: High-priority control parameters for collisions. Data source details are provided on our website.