ISS-Scenario: Scenario-based Testing in CARLA
Renjue Li, Tianhang Qin, Cas Widdershoven
TL;DR
ISS-Scenario tackles the safety-testing bottleneck for autonomous driving systems by enabling scalable, scenario-based evaluation in the CARLA simulator. It introduces a parameterized scenario library, a batch testing workflow with both uniform and GA-driven sampling, and an accident replay feature to reproduce and analyze incidents. Through case studies with TCP and Interfuser, the paper shows that a genetic algorithm can uncover more dangerous scenarios than random sampling, including night-time conditions that raise safety concerns. The work yields a practical, open-source tool for systematic ADS testing and safety validation, reducing the cost and effort of identifying unsafe behaviors in complex traffic.
Abstract
The rapidly evolving field of autonomous driving systems (ADSs) is full of promise. However, in order to fulfil these promises, ADSs need to be safe in all circumstances. This paper introduces ISS-Scenario, an autonomous driving testing framework in the paradigm of scenario-based testing. ISS-Scenario is designed for batch testing, exploration of test cases (e.g., potentially dangerous scenarios), and performance evaluation of autonomous vehicles (AVs). ISS-Scenario includes a diverse simulation scenario library with parametrized design. Furthermore, ISS-Scenario integrates two testing methods within the framework: random sampling and optimized search by means of a genetic algorithm. Finally, ISS-Scenario provides an accident replay feature, saving a log file for each test case which allows developers to replay and dissect scenarios where the ADS showed problematic behavior.