Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dance of the ADS: Orchestrating Failures through Historically-Informed Scenario Fuzzing

Tong Wang, Taotao Gu, Huan Deng, Hu Li, Xiaohui Kuang, Gang Zhao

TL;DR

This work tackles the challenge of safety verification for autonomous driving systems (ADS) in the absence of predefined starting scenarios. It introduces ScenarioFuzz, a map-aware fuzzing framework that builds a seed corpus from road-network data (OPENDRIVE), mutates seeds via a two-stage process with domain-specific mutators, and filters high-risk seeds with a Graph Neural Network-based scenario evaluation model (SEM). A self-supervised collision-trajectory clustering method identifies 54 high-risk collision categories and exposes 58 bugs across six ADS while integrating with the CARLA simulator to enable end-to-end testing. The approach delivers substantial efficiency gains, reducing per-scenario execution time by 60.3% and increasing error-discovery rate by 103% on average, demonstrating practical value for scalable ADS safety testing and validation.

Abstract

As autonomous driving systems (ADS) advance towards higher levels of autonomy, orchestrating their safety verification becomes increasingly intricate. This paper unveils ScenarioFuzz, a pioneering scenario-based fuzz testing methodology. Designed like a choreographer who understands the past performances, it uncovers vulnerabilities in ADS without the crutch of predefined scenarios. Leveraging map road networks, such as OPENDRIVE, we extract essential data to form a foundational scenario seed corpus. This corpus, enriched with pertinent information, provides the necessary boundaries for fuzz testing in the absence of starting scenarios. Our approach integrates specialized mutators and mutation techniques, combined with a graph neural network model, to predict and filter out high-risk scenario seeds, optimizing the fuzzing process using historical test data. Compared to other methods, our approach reduces the time cost by an average of 60.3%, while the number of error scenarios discovered per unit of time increases by 103%. Furthermore, we propose a self-supervised collision trajectory clustering method, which aids in identifying and summarizing 54 high-risk scenario categories prone to inducing ADS faults. Our experiments have successfully uncovered 58 bugs across six tested systems, emphasizing the critical safety concerns of ADS.

Dance of the ADS: Orchestrating Failures through Historically-Informed Scenario Fuzzing

TL;DR

This work tackles the challenge of safety verification for autonomous driving systems (ADS) in the absence of predefined starting scenarios. It introduces ScenarioFuzz, a map-aware fuzzing framework that builds a seed corpus from road-network data (OPENDRIVE), mutates seeds via a two-stage process with domain-specific mutators, and filters high-risk seeds with a Graph Neural Network-based scenario evaluation model (SEM). A self-supervised collision-trajectory clustering method identifies 54 high-risk collision categories and exposes 58 bugs across six ADS while integrating with the CARLA simulator to enable end-to-end testing. The approach delivers substantial efficiency gains, reducing per-scenario execution time by 60.3% and increasing error-discovery rate by 103% on average, demonstrating practical value for scalable ADS safety testing and validation.

Abstract

As autonomous driving systems (ADS) advance towards higher levels of autonomy, orchestrating their safety verification becomes increasingly intricate. This paper unveils ScenarioFuzz, a pioneering scenario-based fuzz testing methodology. Designed like a choreographer who understands the past performances, it uncovers vulnerabilities in ADS without the crutch of predefined scenarios. Leveraging map road networks, such as OPENDRIVE, we extract essential data to form a foundational scenario seed corpus. This corpus, enriched with pertinent information, provides the necessary boundaries for fuzz testing in the absence of starting scenarios. Our approach integrates specialized mutators and mutation techniques, combined with a graph neural network model, to predict and filter out high-risk scenario seeds, optimizing the fuzzing process using historical test data. Compared to other methods, our approach reduces the time cost by an average of 60.3%, while the number of error scenarios discovered per unit of time increases by 103%. Furthermore, we propose a self-supervised collision trajectory clustering method, which aids in identifying and summarizing 54 high-risk scenario categories prone to inducing ADS faults. Our experiments have successfully uncovered 58 bugs across six tested systems, emphasizing the critical safety concerns of ADS.
Paper Structure (25 sections, 15 figures, 3 tables, 2 algorithms)

This paper contains 25 sections, 15 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Autonomous Driving Systems (ADS)
  4. Scenarios in the Context of ADS
  5. Approach
  6. Overview of the ScenarioFuzz Method
  7. Building Corpus with Map Crawling
  8. Scenario Seed Selection
  9. Scenario Seed Mutation
  10. Mutators
  11. Mutation Strategy
  12. Mutated Seeds Filtering
  13. Execution and Result Monitoring
  14. Seed Scheduling Strategy
  15. Replay and Analysis of Test Results
  16. ...and 10 more sections

Figures (15)

  • Figure 1: Autonomous driving system and testing scenarios.
  • Figure 2: Overview of the scenario layer model zhongSurveyScenarioBasedTesting2021.
  • Figure 3: Overview of the architecture and workflow of ScenarioFuzz.
  • Figure 4: The comprehensive process and architectural diagram of the scenario evaluation model.
  • Figure 5: Schematic of the vehicle condition clustering method based on self-supervised learning and SEM features.
  • ...and 10 more figures