Fine-grained Testing for Autonomous Driving Software: a Study on Autoware with LLM-driven Unit Testing

TL;DR

This work explores fine-grained unit testing for autonomous driving software using Autoware as a case study. It first documents that developer-written tests provide limited code coverage, especially in perception and sensing modules, and that even targeted, branched functions often remain untested. It then evaluates LLM-based automatic test generation, finding low build/run success and tepid code coverage due to issues such as symbol errors and hallucinations in function usage. To address this, the authors propose AwTest-LLM, a dependency-aware testing framework that uses AST-based dependency extraction, call graphs for few-shot context, and CMake parsing to generate higher-quality unit tests; improvements are observed on the covered function set, though untested functions continue to pose challenges. Overall, the paper highlights both the potential and the hurdles of integrating LLM-driven unit testing into industrial ADS development, pointing to future work in robust assertion generation and broader ecosystem integration.

Abstract

Testing autonomous driving systems (ADS) is critical to ensuring their reliability and safety. Existing ADS testing works focuses on designing scenarios to evaluate system-level behaviors, while fine-grained testing of ADS source code has received comparatively little attention. To address this gap, we present the first study on testing, specifically unit testing, for ADS source code. Our study focuses on an industrial ADS framework, Autoware. We analyze both human-written test cases and those generated by large language models (LLMs). Our findings reveal that human-written test cases in Autoware exhibit limited test coverage, and significant challenges remain in applying LLM-generated tests for Autoware unit testing. To overcome these challenges, we propose AwTest-LLM, a novel approach to enhance test coverage and improve test case pass rates across Autoware packages.

Figures (1)

• Figure 1: The overview of AwTest-LLM.