ICST Tool Competition 2025 -- Self-Driving Car Testing Track

TL;DR

The paper tackles the problem of efficiently selecting regression tests for simulation-based self-driving car systems, where long-running simulations and non-determinism create high costs. It introduces the ICST Tool Competition 2025 framework, featuring a Protocol Buffers/gRPC interface, Dockerized tools, and the SensoDat benchmark suite with 36 test collections generated by Ambiegen, Frenetic, and FreneticV. Five tools are evaluated against metrics including Selection Count, Initialization Time, Selection Time, $Simulation Time to Fault Ratio$, and $Fault to Selection Ratio$, with a random baseline as control. ITS4SDC emerges as the best-performing tool for cost-effectiveness, revealing the practical viability of automated test selection for SDC regression testing and pointing to future work that adds environmental factors and more diverse evaluation metrics.

Abstract

This is the first edition of the tool competition on testing self-driving cars (SDCs) at the International Conference on Software Testing, Verification and Validation (ICST). The aim is to provide a platform for software testers to submit their tools addressing the test selection problem for simulation-based testing of SDCs, which is considered an emerging and vital domain. The competition provides an advanced software platform and representative case studies to ease participants' entry into SDC regression testing, enabling them to develop their initial test generation tools for SDCS. In this first edition, the competition includes five tools from different authors. All tools were evaluated using (regression) metrics for test selection as well as compared with a baseline approache. This paper provides an overview of the competition, detailing its context, framework, participating tools, evaluation methodology, and key findings.

Figures (2)

• Figure 1: Sequence Diagram of the Tool and Evaluator Intercation
• Figure 2: High-level Platform Abstraction