ADPerf: Investigating and Testing Performance in Autonomous Driving Systems

TL;DR

This work tackles the underexplored problem of obstacle-detection latency in autonomous driving systems and its cascading impact on downstream modules. It combines model-based performance analysis using Queuing Networks and Queueing Petri Nets with an automated testing framework, ADPerf, to systematically stress 3D obstacle detection by mutating LiDAR point clouds. The study identifies 3D obstacle detection as a key bottleneck in both Apollo and Autoware, demonstrates that latency increases can cause frame drops and significantly affect trajectory prediction, and validates these findings with both real-data experiments and simulator-based demonstrations. The proposed ADPerf framework, together with the QPN-based performance model, provides a practical pathway for latency-aware testing and verification of ADS reliability in realistic workloads.

Abstract

Obstacle detection is crucial to the operation of autonomous driving systems, which rely on multiple sensors, such as cameras and LiDARs, combined with code logic and deep learning models to detect obstacles for time-sensitive decisions. Consequently, obstacle detection latency is critical to the safety and effectiveness of autonomous driving systems. However, the latency of the obstacle detection module and its resilience to various changes in the LiDAR point cloud data are not yet fully understood. In this work, we present the first comprehensive investigation on measuring and modeling the performance of the obstacle detection modules in two industry-grade autonomous driving systems, i.e., Apollo and Autoware. Learning from this investigation, we introduce ADPerf, a tool that aims to generate realistic point cloud data test cases that can expose increased detection latency. Increasing latency decreases the availability of the detected obstacles and stresses the capabilities of subsequent modules in autonomous driving systems, i.e., the modules may be negatively impacted by the increased latency in obstacle detection. We applied ADPerf to stress-test the performance of widely used 3D obstacle detection modules in autonomous driving systems, as well as the propagation of such tests on trajectory prediction modules. Our evaluation highlights the need to conduct performance testing of obstacle detection components, especially 3D obstacle detection, as they can be a major bottleneck to increased latency of the autonomous driving system. Such an adverse outcome will also further propagate to other modules, reducing the overall reliability of autonomous driving systems.

Figures (9)

• Figure 1: The data flow between the modules (with a focus on sensing and perception components) in a general ADS. Arrows indicate data channels.
• Figure 2: Overview of our experiment
• Figure 3: Performance modeling of Apollo obstacle detection. Modeling figures for Autoware are included in the artifact.
• Figure 4: Measured latency of the 3D obstacle detection modules of two ADSs. Latencies are grouped into bins by the number of detections, with the percentage of frames in each bin indicated. Violin plots with marked medians illustrate the latency distribution within each bin. Regression lines reveal a positive correlation between detection count and latency, and are further used to predict latency for unseen frames.
• Figure 5: Overview of the ADPerf, showing how the data flows from the beginning (nuScenes).