Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SimADFuzz: Simulation-Feedback Fuzz Testing for Autonomous Driving Systems

Huiwen Yang, Yu Zhou, Taolue Chen

TL;DR

SimADFuzz tackles the challenge of robustly testing autonomous driving systems by integrating simulation feedback with a model-based, temporal-aware fitness evaluation and a distance-guided mutation strategy. It leverages a Transformer encoder to embed driving scenarios into a high-dimensional temporal representation and uses a violation-prediction layer within NSGA-2 to prioritize high-risk scenarios, while its mutation strategy actively increases interactions between actors. Experimental results on InterFuser in CARLA Town03 show that SimADFuzz discovers significantly more unique violations and achieves much higher ego-trajectory diversity than state-of-the-art fuzzers, including reproducing several collisions. This approach enhances ADS safety assessment by producing richer, more actionable test scenarios and offers a reproducible framework for future simulation-based robustness testing.

Abstract

Autonomous driving systems (ADS) have achieved remarkable progress in recent years. However, ensuring their safety and reliability remains a critical challenge due to the complexity and uncertainty of driving scenarios. In this paper, we focus on simulation testing for ADS, where generating diverse and effective testing scenarios is a central task. Existing fuzz testing methods face limitations, such as overlooking the temporal and spatial dynamics of scenarios and failing to leverage simulation feedback (e.g., speed, acceleration and heading) to guide scenario selection and mutation. To address these issues, we propose SimADFuzz, a novel framework designed to generate high-quality scenarios that reveal violations in ADS behavior. Specifically, SimADFuzz employs violation prediction models, which evaluate the likelihood of ADS violations, to optimize scenario selection. Moreover, SimADFuzz proposes distance-guided mutation strategies to enhance interactions among vehicles in offspring scenarios, thereby triggering more edge-case behaviors of vehicles. Comprehensive experiments demonstrate that SimADFuzz outperforms state-of-the-art fuzzers by identifying 32 more unique violations, including 4 reproducible cases of vehicle-vehicle and vehicle-pedestrian collisions. These results demonstrate SimADFuzz's effectiveness in enhancing the robustness and safety of autonomous driving systems.

SimADFuzz: Simulation-Feedback Fuzz Testing for Autonomous Driving Systems

TL;DR

SimADFuzz tackles the challenge of robustly testing autonomous driving systems by integrating simulation feedback with a model-based, temporal-aware fitness evaluation and a distance-guided mutation strategy. It leverages a Transformer encoder to embed driving scenarios into a high-dimensional temporal representation and uses a violation-prediction layer within NSGA-2 to prioritize high-risk scenarios, while its mutation strategy actively increases interactions between actors. Experimental results on InterFuser in CARLA Town03 show that SimADFuzz discovers significantly more unique violations and achieves much higher ego-trajectory diversity than state-of-the-art fuzzers, including reproducing several collisions. This approach enhances ADS safety assessment by producing richer, more actionable test scenarios and offers a reproducible framework for future simulation-based robustness testing.

Abstract

Autonomous driving systems (ADS) have achieved remarkable progress in recent years. However, ensuring their safety and reliability remains a critical challenge due to the complexity and uncertainty of driving scenarios. In this paper, we focus on simulation testing for ADS, where generating diverse and effective testing scenarios is a central task. Existing fuzz testing methods face limitations, such as overlooking the temporal and spatial dynamics of scenarios and failing to leverage simulation feedback (e.g., speed, acceleration and heading) to guide scenario selection and mutation. To address these issues, we propose SimADFuzz, a novel framework designed to generate high-quality scenarios that reveal violations in ADS behavior. Specifically, SimADFuzz employs violation prediction models, which evaluate the likelihood of ADS violations, to optimize scenario selection. Moreover, SimADFuzz proposes distance-guided mutation strategies to enhance interactions among vehicles in offspring scenarios, thereby triggering more edge-case behaviors of vehicles. Comprehensive experiments demonstrate that SimADFuzz outperforms state-of-the-art fuzzers by identifying 32 more unique violations, including 4 reproducible cases of vehicle-vehicle and vehicle-pedestrian collisions. These results demonstrate SimADFuzz's effectiveness in enhancing the robustness and safety of autonomous driving systems.

Paper Structure

This paper contains 27 sections, 1 equation, 13 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. A Motivating Example
  4. Approach
  5. Simulation Test Engine and Feedback Collection
  6. Simulation-Feedback Genetic Algorithm
  7. Model-based Fitness Evaluation and Scenario Selection
  8. Crossover Strategy
  9. Mutation Strategy
  10. Violation Detector and Reproduction
  11. Evaluation and Results
  12. Experimental Settings
  13. ADS Under Test
  14. Fuzzing Configurations
  15. Baselines and Evaluation Metrics
  16. ...and 12 more sections

Figures (13)

  • Figure 1: Simulation-based Fuzz Testing Framework for ADS
  • Figure 2: Multi-Vehicle Interaction at a T-junction
  • Figure 3: Overview of SimADFuzz
  • Figure 4: Process of the Simulation-Feedback Genetic Algorithm
  • Figure 5: An Example of Distance-guided Mutation
  • ...and 8 more figures