Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Perception-Guided Fuzzing for Simulated Scenario-Based Testing of Autonomous Driving Systems

Tri Minh Triet Pham, Bo Yang, Jinqiu Yang

TL;DR

This paper tackles the challenge of safely testing autonomous driving systems by focusing on the perception module within a system-level framework. It introduces SimsV, a perception-guided fuzzing approach that uses a high-fidelity simulator to mutate driving scenes and collect runtime perception metrics as feedback, enabling grey-box, mutation-driven test generation. The method is demonstrated on Apollo (L4) with LGSVL, revealing perceptual weaknesses and revealing cases that lead to collisions, unnecessary stops, or incorrect destinations. The work contributes a practical framework for exposure of perception faults, guides test generation with neuron-activation and undetected-object metrics, and offers data-driven insights to improve ADS safety and reliability in real-world deployment.

Abstract

Autonomous Driving Systems (ADS) have made huge progress and started on-road testing or even commercializing trials. ADS are complex and difficult to test: they receive input data from multiple sensors and make decisions using a combination of multiple deep neural network models and code logic. The safety of ADS is of utmost importance as their misbehavior can result in costly catastrophes, including the loss of human life. In this work, we propose SimsV, which performs system-level testing on multi-module ADS. SimsV targets perception failures of ADS and further assesses the impact of perception failure on the system as a whole. SimsV leverages a high-fidelity simulator for test input and oracle generation by continuously applying predefined mutation operators. In addition, SimsV leverages various metrics to guide the testing process. We implemented a prototype SimsV for testing a commercial-grade Level 4 ADS (i.e., Apollo) using a popular open-source driving platform simulator. Our evaluation shows that SimsV is capable of finding weaknesses in the perception of Apollo. Furthermore, we show that by exploiting such weakness, SimsV finds severe problems in Apollo, including collisions.

Perception-Guided Fuzzing for Simulated Scenario-Based Testing of Autonomous Driving Systems

TL;DR

This paper tackles the challenge of safely testing autonomous driving systems by focusing on the perception module within a system-level framework. It introduces SimsV, a perception-guided fuzzing approach that uses a high-fidelity simulator to mutate driving scenes and collect runtime perception metrics as feedback, enabling grey-box, mutation-driven test generation. The method is demonstrated on Apollo (L4) with LGSVL, revealing perceptual weaknesses and revealing cases that lead to collisions, unnecessary stops, or incorrect destinations. The work contributes a practical framework for exposure of perception faults, guides test generation with neuron-activation and undetected-object metrics, and offers data-driven insights to improve ADS safety and reliability in real-world deployment.

Abstract

Autonomous Driving Systems (ADS) have made huge progress and started on-road testing or even commercializing trials. ADS are complex and difficult to test: they receive input data from multiple sensors and make decisions using a combination of multiple deep neural network models and code logic. The safety of ADS is of utmost importance as their misbehavior can result in costly catastrophes, including the loss of human life. In this work, we propose SimsV, which performs system-level testing on multi-module ADS. SimsV targets perception failures of ADS and further assesses the impact of perception failure on the system as a whole. SimsV leverages a high-fidelity simulator for test input and oracle generation by continuously applying predefined mutation operators. In addition, SimsV leverages various metrics to guide the testing process. We implemented a prototype SimsV for testing a commercial-grade Level 4 ADS (i.e., Apollo) using a popular open-source driving platform simulator. Our evaluation shows that SimsV is capable of finding weaknesses in the perception of Apollo. Furthermore, we show that by exploiting such weakness, SimsV finds severe problems in Apollo, including collisions.
Paper Structure (14 sections, 13 figures, 1 table, 1 algorithm)

This paper contains 14 sections, 13 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Related Work
  4. SimsV: Methodology and Implementation
  5. An Overview of SimsV
  6. Data Extraction from ADS
  7. Perception Evaluation
  8. Mutation-based scene generation
  9. Evaluation
  10. Threats to validity
  11. Internal Validity
  12. External Validity
  13. Discussions and Future Work
  14. Conclusions

Figures (13)

  • Figure 1: An overview of a multi-module ADS.
  • Figure 2: Obstacle Detection in the perception module of Apollo 7.0
  • Figure 3: An overview of SimsV
  • Figure 4: SimsV compares the obstacle detection results from Apollo with the ground truth. Enlarged and highlighted in green is an abstracted message received from detection, which includes the detected obstacles and their attributes (i.e., the position of the center, velocity, etc.). Three windows are displayed in the figure. From left to right are windows showing (1) obstacle detection messages, (2) obstacles detected visualized in Dreamview, and (3) obstacles as they appear in LGSVL.
  • Figure 5: Average precision for 300 rounds by SimsV (neuron-coverage guided) for camera-LiDAR fusion (the top row), solo LiDAR (the middle row), and solo camera (the bottom row).
  • ...and 8 more figures