Hi-ALPS -- An Experimental Robustness Quantification of Six LiDAR-based Object Detection Systems for Autonomous Driving
Alexandra Arzberger, Ramin Tavakoli Kolagari
TL;DR
The paper tackles robustness of LiDAR-based 3D object detection in autonomous driving. It introduces Hi-ALPS, a hierarchical perturbation level system, implemented via the AELiDAR framework to generate perturbations in a black-box setting; it evaluates six state-of-the-art detectors on the KITTI dataset and reports that none remain robust beyond Hi-ALPS level 1, with $mAP$ dropping to around $75\%$ at level 2 for some perturbations and below $50\%$ at level 5 in others. It discusses perceptibility, real-world plausibility, and potential countermeasures, arguing that robustness hinges on better training data diversity and targeted defenses. The work provides a structured robustness benchmark for LiDAR OD and highlights the need for data-centric improvements before deploying complex adversarial defenses.
Abstract
Light Detection and Ranging (LiDAR) is an essential sensor technology for autonomous driving as it can capture high-resolution 3D data. As 3D object detection systems (OD) can interpret such point cloud data, they play a key role in the driving decisions of autonomous vehicles. Consequently, such 3D OD must be robust against all types of perturbations and must therefore be extensively tested. One approach is the use of adversarial examples, which are small, sometimes sophisticated perturbations in the input data that change, i.e., falsify, the prediction of the OD. These perturbations are carefully designed based on the weaknesses of the OD. The robustness of the OD cannot be quantified with adversarial examples in general, because if the OD is vulnerable to a given attack, it is unclear whether this is due to the robustness of the OD or whether the attack algorithm produces particularly strong adversarial examples. The contribution of this work is Hi-ALPS -- Hierarchical Adversarial-example-based LiDAR Perturbation Level System, where higher robustness of the OD is required to withstand the perturbations as the perturbation levels increase. In doing so, the Hi-ALPS levels successively implement a heuristic followed by established adversarial example approaches. In a series of comprehensive experiments using Hi-ALPS, we quantify the robustness of six state-of-the-art 3D OD under different types of perturbations. The results of the experiments show that none of the OD is robust against all Hi-ALPS levels; an important factor for the ranking is that human observers can still correctly recognize the perturbed objects, as the respective perturbations are small. To increase the robustness of the OD, we discuss the applicability of state-of-the-art countermeasures. In addition, we derive further suggestions for countermeasures based on our experimental results.