SafePlanner: Testing Safety of the Automated Driving System Plan Model

TL;DR

SafePlanner addresses the safety evaluation gap in autonomous driving systems by focusing on the Plan model within the Sense-Plan-Act stack. It introduces a white-box, scenario-driven testing framework that decomposes driving scenarios into scene transitions and NPC behaviors, guided by a mutation-based fuzzer and a fitness function targeting dynamic driving tasks. The approach yields 520 hazardous behaviors across 20,635 test cases, organized into 15 root causes, with 4 issues patched and improvements in function and decision coverage. These results demonstrate the value of structured, planner-aware testing for ADS safety and support broader adoption of white-box methodologies in automated driving evaluation. The work has practical impact by enabling more systematic detection and localization of planning flaws in production-grade ADS like Baidu Apollo.

Abstract

In this work, we present SafePlanner, a systematic testing framework for identifying safety-critical flaws in the Plan model of Automated Driving Systems (ADS). SafePlanner targets two core challenges: generating structurally meaningful test scenarios and detecting hazardous planning behaviors. To maximize coverage, SafePlanner performs a structural analysis of the Plan model implementation - specifically, its scene-transition logic and hierarchical control flow - and uses this insight to extract feasible scene transitions from code. It then composes test scenarios by combining these transitions with non-player vehicle (NPC) behaviors. Guided fuzzing is applied to explore the behavioral space of the Plan model under these scenarios. We evaluate SafePlanner on Baidu Apollo, a production-grade level 4 ADS. It generates 20635 test cases and detects 520 hazardous behaviors, grouped into 15 root causes through manual analysis. For four of these, we applied patches based on our analysis; the issues disappeared, and no apparent side effects were observed. SafePlanner achieves 83.63 percent function and 63.22 percent decision coverage on the Plan model, outperforming baselines in both bug discovery and efficiency.

Figures (6)

• Figure 1: The Sense-Plan-Act model of ADS
• Figure 2: Overview of SafePlanner
• Figure 3: Location and maneuver of NPC behavior
• Figure 4: Illustration of discovered hazardous behaviors
• Figure 5: Code coverage result of Baidu Apollo Plan model