SoVAR: Building Generalizable Scenarios from Accident Reports for Autonomous Driving Testing
An Guo, Yuan Zhou, Haoxiang Tian, Chunrong Fang, Yunjian Sun, Weisong Sun, Xinyu Gao, Anh Tuan Luu, Yang Liu, Zhenyu Chen
TL;DR
SoVAR introduces an automatic pipeline that converts textual accident reports into road-generalizable accident scenarios for autonomous driving testing. It uses carefully designed linguistic-pattern prompts with GPT-4 to extract environmental, road, and dynamic-object information, then solves trajectory constraints with a solver to generate crash trajectories that fit different map structures. The reconstructed scenarios are transformed into ADS test cases and evaluated on Baidu Apollo using LGSVL/SORA-SVL, demonstrating strong information extraction accuracy and generalization across road types, and uncovering several safety-violation scenarios. This approach enables targeted, map-robust ADS testing and provides a scalable method to translate real-world crashes into actionable simulation scenarios with practical implications for improving ADS safety.
Abstract
Autonomous driving systems (ADSs) have undergone remarkable development and are increasingly employed in safety-critical applications. However, recently reported data on fatal accidents involving ADSs suggests that the desired level of safety has not yet been fully achieved. Consequently, there is a growing need for more comprehensive and targeted testing approaches to ensure safe driving. Scenarios from real-world accident reports provide valuable resources for ADS testing, including critical scenarios and high-quality seeds. However, existing scenario reconstruction methods from accident reports often exhibit limited accuracy in information extraction. Moreover, due to the diversity and complexity of road environments, matching current accident information with the simulation map data for reconstruction poses significant challenges. In this paper, we design and implement SoVAR, a tool for automatically generating road-generalizable scenarios from accident reports. SoVAR utilizes well-designed prompts with linguistic patterns to guide the large language model in extracting accident information from textual data. Subsequently, it formulates and solves accident-related constraints in conjunction with the extracted accident information to generate accident trajectories. Finally, SoVAR reconstructs accident scenarios on various map structures and converts them into test scenarios to evaluate its capability to detect defects in industrial ADSs. We experiment with SoVAR, using accident reports from the National Highway Traffic Safety Administration's database to generate test scenarios for the industrial-grade ADS Apollo. The experimental findings demonstrate that SoVAR can effectively generate generalized accident scenarios across different road structures. Furthermore, the results confirm that SoVAR identified 5 distinct safety violation types that contributed to the crash of Baidu Apollo.