Spotting the Unexpected (STU): A 3D LiDAR Dataset for Anomaly Segmentation in Autonomous Driving

TL;DR

Spotting the Unexpected (STU) addresses the lack of real-world 3D anomaly segmentation data by introducing a high-resolution LiDAR–camera, surround-view dataset with dense semantic and instance labels for anomalies and inliers. The authors adapt Mask4Former-3D and establish baselines (including Deep Ensembles, MC Dropout, and a Void Classifier) to benchmark anomaly segmentation in 3D, reporting clear gaps between 2D and 3D methods and highlighting the challenge of detecting sparse, distant anomalies. They provide a thorough dataset analysis, post-processing and ground-plane techniques, and a detailed training protocol, including a dedicated STU-inlier set to reduce domain gap. The work enables standardized evaluation for multimodal, temporal 3D anomaly segmentation and sets the stage for improved open-set 3D perception in autonomous driving, with practical implications for safer navigation and validation. Future work is directed at leveraging temporal information and more sophisticated multimodal models to bridge the 3D anomaly detection gap.

Abstract

To operate safely, autonomous vehicles (AVs) need to detect and handle unexpected objects or anomalies on the road. While significant research exists for anomaly detection and segmentation in 2D, research progress in 3D is underexplored. Existing datasets lack high-quality multimodal data that are typically found in AVs. This paper presents a novel dataset for anomaly segmentation in driving scenarios. To the best of our knowledge, it is the first publicly available dataset focused on road anomaly segmentation with dense 3D semantic labeling, incorporating both LiDAR and camera data, as well as sequential information to enable anomaly detection across various ranges. This capability is critical for the safe navigation of autonomous vehicles. We adapted and evaluated several baseline models for 3D segmentation, highlighting the challenges of 3D anomaly detection in driving environments. Our dataset and evaluation code will be openly available, facilitating the testing and performance comparison of different approaches.

Figures (20)

• Figure 1: We present Spotting the Unexpected (STU) a novel anomaly segmentation dataset for autonomous driving. The dataset contains semantic and instance labels for out-of-distribution (OOD) objects, and includes surround-view setup with synchronized cameras.
• Figure 2: Data collection conducted in a naturalistic manner (a) and controlled environment (b) with objects on the road.
• Figure 3: Different anomalies in the STU dataset. Different objects on the road used for staged data collection. We pick objects such that have no intersection with the inlier dataset and place them on roads in different locations and illumination conditions. Objects might touch each other, be very small, as large as a chair or a surf board, and could cause an accident if a car would drive over them.
• Figure 4: Anomaly Instance Properties. A typical recorded anomaly has on average less then $50$ points per sequence (a), with less then $300$ (b) points at maximum, and a maximum height below one meter (c). We record up to nine individual anomaly instances in the same sequence (d).
• Figure 5: Distribution of anomalies along the vehicle's reference frame. Most of the points appear around the vehicle.