Dual-level Adaptive Self-Labeling for Novel Class Discovery in Point Cloud Segmentation

TL;DR

This work tackles novel class discovery in 3D point cloud segmentation under open-world conditions, where novel classes are imbalanced and spatial context is crucial. It proposes a dual-level adaptive self-labeling framework that combines point- and region-level representations with a prototype-based classifier, and generates high-quality imbalanced pseudo-labels via a semi-relaxed Optimal Transport objective guided by an adaptive KL-based regularization. The method learns through an iterative process that updates pseudo-labels and the model, with an indicator-driven hyperparameter search to balance known- and novel-class learning. Empirical results on SemanticKITTI and SemanticPOSS show substantial gains over prior approaches, validating the effectiveness of regional context and adaptive pseudo-labeling for open-world 3D perception.

Abstract

We tackle the novel class discovery in point cloud segmentation, which discovers novel classes based on the semantic knowledge of seen classes. Existing work proposes an online point-wise clustering method with a simplified equal class-size constraint on the novel classes to avoid degenerate solutions. However, the inherent imbalanced distribution of novel classes in point clouds typically violates the equal class-size constraint. Moreover, point-wise clustering ignores the rich spatial context information of objects, which results in less expressive representation for semantic segmentation. To address the above challenges, we propose a novel self-labeling strategy that adaptively generates high-quality pseudo-labels for imbalanced classes during model training. In addition, we develop a dual-level representation that incorporates regional consistency into the point-level classifier learning, reducing noise in generated segmentation. Finally, we conduct extensive experiments on two widely used datasets, SemanticKITTI and SemanticPOSS, and the results show our method outperforms the state of the art by a large margin.

Figures (7)

• Figure 1: Our method starts with two views of the same point cloud ($x$ and $\hat{x}$) and clusters the points into corresponding regions. Then, we extract individual point features via a forward pass and calculate regional representations by averaging the point features within each region. Next, we make predictions $p$ by the classifier $h$, and generate pseudo labels $y$ for unlabeled points and regions using our novel adaptive self-labeling algorithm (generating pseudo-labels does not involve gradients). Lastly, we exchange the pseudo labels between the two views and update the model accordingly.
• Figure 2: Visualization comparison between Our method and NOPS on the SemanticPOSS and SemanticKITTI datasets. In the first and second rows, compared to NOPS, our method achieves much better segmentation for the 'Building', significantly reducing confusion with tail classes (such as 'Car') or medium classes (like 'Plants'). In the third row, our approach generates better segmentation for 'Parking' and 'Car'.
• Figure 3: Confusion Matrix, GT on the y-axis, Pseudo Label on the x-axis. $(i, j)$ represents the % of GT in class $j$ assigned pseudo label $i$. We categorize 'plants' and 'ground' as head classes, 'building' as medium, and 'car' as tail classes.
• Figure 4: Visualization analysis. The introduction of ISL notably reduces the misclassification between 'Plant' and 'Car'. Then, the integration of AR further mitigates the confusion between 'Plant' and 'Building'. Ultimately, the incorporation of Region component (Ours) effectively minimizes the mix-up between 'Plant', 'Car', and 'Building'.
• Figure 5: $\gamma$ variation