PDF: A Probability-Driven Framework for Open World 3D Point Cloud Semantic Segmentation

TL;DR

The paper tackles open-world semantic segmentation for 3D point clouds by introducing the Probability-Driven Framework (PDF), which simultaneously identifies unknown objects and enables continual learning. PDF combines a lightweight U-decoder to estimate uncertainties (O_U) with a semantic decoder (O_S), uses a pseudo-labeling scheme to create pseudo GT for unknowns, and employs incremental knowledge distillation to integrate novel classes without retraining from scratch. Key contributions include the OSS module with uncertainty-aware supervision, the HUA and 3D graph boundary detection pipeline for refining unknown regions, and the IL strategy that distills knowledge from a teacher model to a new open-world model. Experiments on S3DIS and ScanNetv2 show that PDF substantially improves unknown-object identification and achieves strong incremental learning performance, outperforming prior OWSS methods. The work advances practical open-world perception for 3D scenes with implications for robotics and autonomous systems.

Abstract

Existing point cloud semantic segmentation networks cannot identify unknown classes and update their knowledge, due to a closed-set and static perspective of the real world, which would induce the intelligent agent to make bad decisions. To address this problem, we propose a Probability-Driven Framework (PDF) for open world semantic segmentation that includes (i) a lightweight U-decoder branch to identify unknown classes by estimating the uncertainties, (ii) a flexible pseudo-labeling scheme to supply geometry features along with probability distribution features of unknown classes by generating pseudo labels, and (iii) an incremental knowledge distillation strategy to incorporate novel classes into the existing knowledge base gradually. Our framework enables the model to behave like human beings, which could recognize unknown objects and incrementally learn them with the corresponding knowledge. Experimental results on the S3DIS and ScanNetv2 datasets demonstrate that the proposed PDF outperforms other methods by a large margin in both important tasks of open world semantic segmentation.

Figures (3)

• Figure 1: The closed-set model $\mathcal{M}_C$ continuously improves its open-world capabilities by successively finetuning to open-set model $\mathcal{M}_O$ and open-world $\mathcal{M}_I$ with the help of open-set semantic segmentation (OSS) task and incremental learning (IL) task, where the proposed pseudo-labeling scheme and incremental knowledge distillation strategy are employed, respectively.
• Figure 2: Edge weights distribution. The edges' weights between nodes of known classes are distinct from unknown classes. The distribution is approximately fitted with a Gaussian mixed model, and divided by the threshold $\mu_1 - \epsilon \sigma_1$ derived from the 3$\sigma$ criteria.
• Figure 3: Comparing the open-set semantic segmentation (OSS) results of our method (g) and other OSS methods (b-f).