Label-Free Model Failure Detection for Lidar-based Point Cloud Segmentation

TL;DR

This paper tackles the inadequacy of small labeled evaluation sets for revealing failures in lidar-based point cloud segmentation. It introduces a label-free failure-detection framework based on complementary learning: a supervised stream yields semantic motion labels and a self-supervised stream yields predictive motion labels, with per-point discrepancies used to flag potential failures. The authors also present LidarCODA, the first public real-world lidar anomaly dataset with point-wise 3D labels, enabling quantitative analysis of failure sensitivity to outliers. Experiments on regular and anomalous scenarios show the approach uncovers failures beyond what small validation sets reveal, supporting better deployment and data-collection strategies for large-scale autonomous driving systems.

Abstract

Autonomous vehicles drive millions of miles on the road each year. Under such circumstances, deployed machine learning models are prone to failure both in seemingly normal situations and in the presence of outliers. However, in the training phase, they are only evaluated on small validation and test sets, which are unable to reveal model failures due to their limited scenario coverage. While it is difficult and expensive to acquire large and representative labeled datasets for evaluation, large-scale unlabeled datasets are typically available. In this work, we introduce label-free model failure detection for lidar-based point cloud segmentation, taking advantage of the abundance of unlabeled data available. We leverage different data characteristics by training a supervised and self-supervised stream for the same task to detect failure modes. We perform a large-scale qualitative analysis and present LidarCODA, the first publicly available dataset with labeled anomalies in real-world lidar data, for an extensive quantitative analysis.

Figures (10)

• Figure 1: Model Failure Detection. The top left point cloud shows a supervised and the top right a self-supervised motion segmentation. The supervised model falsely classifies the pedestrian in the front left as static. Our approach exposes this model failure, as highlighted in red in the bottom image. More details on the color scheme can be found in Sec. \ref{['subsec:discrepancy']}.
• Figure 2: Overview. Given point clouds, we derive semantic motion labels in a supervised (sv) fashion. In addition, we perform ground segmentation and derive predictive motion labels in a self-supervised (ssv) fashion. Subsequently, we perform point-wise discrepancy detection and classify potential model failures.
• Figure 3: Supervised Semantic Motion Labels. The left semantic segmentation allows no distinction between the parked car at the bottom left and the moving car at the top right. The middle image shows a motion segmentation, where the parked car was classified as static, and the moving car as dynamic. Finally, the right image shows the fused semantic motion labels. Adapted from Sartoris_Anomaly_2022_BA.
• Figure 4: Self-Supervised Predictive Motion Labels. The first image shows the original point cloud in green and the point cloud transformed by the scene flow model and compensated by the ego-motion in red. The second and third images show the result of the spatial and flow-based clustering, respectively. The fourth image shows the final predictive motion labels, with dynamic points in red and static points in green. The scenes are shown from a Bird's-Eye View (BEV) perspective. Adapted from Sartoris_Anomaly_2022_BA.
• Figure 5: Self-Supervised Label Generation. The left graph shows that the magnitude of point-wise flow vectors is insufficient to distinguish between dynamic and static points. The boxplot on the right shows that the normalized standard deviation per instance is significantly lower for dynamic instances. Reprinted from Sartoris_Anomaly_2022_BA.