On the Robustness Evaluation of 3D Obstacle Detection Against Specifications in Autonomous Driving

TL;DR

This work tackles the robustness of LiDAR-based 3D obstacle detection in autonomous driving against specification-based perturbations derived from official LiDAR sensor specs and obstacle-material effects. It introduces SORBET, a framework that injects perturbations into point clouds, quantifies detection deviations, and assesses cascading impacts on trajectory prediction for both academic models and an industry-grade ADS (Apollo). Key contributions include a principled set of perturbations, end-to-end evaluation of detection and prediction, and an analysis of retraining to mitigate perturbation effects, showing that small perturbations can propagate to substantial planning-relevant errors. The findings motivate robustness testing as a practical requirement in ADS development, highlighting architecture-dependent sensitivity and the potential safety implications of perturbation propagation in real-world settings.

Abstract

Autonomous driving systems (ADSs) rely on real-time sensor data, such as cameras and LiDARs, for time-critical decisions using deep neural networks. The accuracy of these decisions is crucial for the widespread adoption of ADSs, as errors can have serious consequences. 3D obstacle detection, in particular, is sensitive to point cloud data (PCD) noise from various sources. However, the robustness of current 3D obstacle detection models against specification-based perturbations remains unevaluated. These perturbations are derived from the specification of LiDAR sensors and previous research on LiDAR's ability to capture objects of different colors and materials. They can manifest as very subtle sensor-based noises or obstacle-specific perturbations. Hence, we propose SORBET, a framework that tests the robustness of 3D obstacle detection models in ADS against such perturbations to the PCD to evaluate their robustness. We applied SORBET to evaluate the robustness of five classic 3D obstacle detection models, including one from an industry-grade Level 4 ADS (Baidu's Apollo). Furthermore, we studied how the deviated obstacle detection results would propagate and negatively impact trajectory prediction. Our evaluation emphasizes the importance of testing 3D obstacle detection against specification-based perturbations. We find that even very subtle changes in the PCD (i.e., removing two points) may introduce a non-trivial decrease in the detection performance. Furthermore, such a negative impact will further propagate to other modules and endanger the safety of the ADS.

Figures (6)

• Figure 1: An Overview of a typical ADS
• Figure 2: Overview of the SORBET framework with numbered steps indicating the dataflow order.
• Figure 3: Left: Original PCD from round 19 of KITTI. Right: perturbed PCD using global distance inaccuracy where each point is moved less than 2cm away from its original position.
• Figure 4: Deviations in x, y, and IoU metrics for detected obstacles by (a) PointPillar and (b) Apollo when there is range inaccuracy (global). Only x, y deviations greater than $\pm$0.5 m and IoU deviations greater than $\pm$0.5 (PointPillar) or $\pm$1 (Apollo) are shown. The black car at (0, 0) shows the location of the ego car.
• Figure 5: Deviations in x, y, and IoU metrics for detected obstacles by PointPillar with decreased reflectivity perturbations. Only x, y deviations greater than $\pm$0.5cm and IoU deviations greater than $\pm$0.5 are shown. The black car at (0, 0) shows the location of the ego car.