A Novel Decomposed Feature-Oriented Framework for Open-Set Semantic Segmentation on LiDAR Data

TL;DR

This paper addresses open-set semantic segmentation on LiDAR data by separating CSS for known classes from anomaly-based detection of unknown objects. It introduces DOSS, a decomposed framework with a cylindrical encoder and dual decoders that produce dedicated features for CSS and anomaly detection, guided by a five-term multi-objective loss and threshold-based anomaly scoring in feature space. The approach pushes known-class features to the surface of a hypersphere while clustering unknown features toward the center, enabling robust OSS via maximum-logit based anomaly detection integrated with CSS. Evaluations on nuScenes and SemanticKITTI show state-of-the-art OSS performance while preserving close-set segmentation accuracy, and the authors provide public code for reproducibility. This method has practical impact for autonomous systems by improving safety and reliability in environments with unknown objects.

Abstract

Semantic segmentation is a key technique that enables mobile robots to understand and navigate surrounding environments autonomously. However, most existing works focus on segmenting known objects, overlooking the identification of unknown classes, which is common in real-world applications. In this paper, we propose a feature-oriented framework for open-set semantic segmentation on LiDAR data, capable of identifying unknown objects while retaining the ability to classify known ones. We design a decomposed dual-decoder network to simultaneously perform closed-set semantic segmentation and generate distinctive features for unknown objects. The network is trained with multi-objective loss functions to capture the characteristics of known and unknown objects. Using the extracted features, we introduce an anomaly detection mechanism to identify unknown objects. By integrating the results of close-set semantic segmentation and anomaly detection, we achieve effective feature-driven LiDAR open-set semantic segmentation. Evaluations on both SemanticKITTI and nuScenes datasets demonstrate that our proposed framework significantly outperforms state-of-the-art methods. The source code will be made publicly available at https://github.com/nubot-nudt/DOSS.

Figures (4)

• Figure 1: Visualization of open-set semantic segmentation. Close-set segmentation (CSS) (top right) only predicts the known classes while recognizing the unknown construction vehicle in the blue ellipses as other known classes. Our method realizes anomaly detection (bottom left), i.e., segments unknown objects, and keeps the ability of CSS. Combining the two results above, we can finally achieve open-set semantic segmentation on LiDAR data (bottom right).
• Figure 2: Framework Overview: We first project points to the cylindrical voxels and extract the point-wise feature from the raw point cloud in the cylindrical encoder, so as to obtain aggregated voxel-wise features. These voxel features are fed to the dual decoders, i.e., semantic decoder and open-set decoder, generating distinct voxel features for guiding the known classes CSS and the anomaly detection of unknown objects. The close-set semantic results and the detected unknown objects are finally combined to realize effective open-set segmentation.
• Figure 3: Voxel features distribution generated by our designed open-set decoder. (a) The expected feature distribution: The faint blue region represents the surface of the hypersphere in high-dimensional feature space. Points on the hypersphere's surface with different colors correspond to different known classes. The red-circled points indicate the mean features of these classes. Yellow points at the center of the hypersphere represent features of unknown objects. The object-sphere loss clusters known class features on the hypersphere's surface and pushes unknown object features toward the center, creating a distinct separation between known and unknown features. The contrastive loss and center loss further cluster and tighten features of the same known class while separating them from different ones, reinforcing the gap between known classes and unknown objects. (b) The tSNE visualization of the actual feature distribution generated by our open-set decoder with one LiDAR frame. Each color indicates one class. Unknown class features (yellow points) are gathered in the center and far from that of known classes. The gaps between known classes are also obvious.
• Figure 4: Visualization of OSS results in nuScenes dataset (first two rows) and SemanticKITTI (the last two rows). Red points belong to the unknown objects (circled with dark green dotted lines), i.e., barrier in the first row, construction vehicle in the second row, and other-vehicle in the last two rows in these cases. As shown in the figures, REAL and ContMAV can not detect all unknown points. Especially, ContMAV regards many points of known classes as unknown ones. Our method manages to segment the complete unknown object with few under-segmentation.