Refining Segmentation On-the-Fly: An Interactive Framework for Point Cloud Semantic Segmentation

TL;DR

This work addresses the challenge of semantic segmentation for entire point-cloud scenes under user guidance by introducing InterPCSeg, an interactive framework that operates on-the-fly with off-the-shelf networks. It treats user corrections as sparse test-time supervision and adds a stabilization energy to ensure stable refinement, while a novel interaction simulator enables objective evaluation. The method combines BN warm-up, a correction energy and a stabilization energy in a test-time loss to refine segmentation with few clicks, and re-infers to produce improved labels. Empirical results on S3DIS and ScanNet show substantial mIoU gains with modest interaction budgets, demonstrating practical impact for rapid scene annotation without offline re-training.

Abstract

Existing interactive point cloud segmentation approaches primarily focus on the object segmentation, which aim to determine which points belong to the object of interest guided by user interactions. This paper concentrates on an unexplored yet meaningful task, i.e., interactive point cloud semantic segmentation, which assigns high-quality semantic labels to all points in a scene with user corrective clicks. Concretely, we presents the first interactive framework for point cloud semantic segmentation, named InterPCSeg, which seamlessly integrates with off-the-shelf semantic segmentation networks without offline re-training, enabling it to run in an on-the-fly manner. To achieve online refinement, we treat user interactions as sparse training examples during the test-time. To address the instability caused by the sparse supervision, we design a stabilization energy to regulate the test-time training process. For objective and reproducible evaluation, we develop an interaction simulation scheme tailored for the interactive point cloud semantic segmentation task. We evaluate our framework on the S3DIS and ScanNet datasets with off-the-shelf segmentation networks, incorporating interactions from both the proposed interaction simulator and real users. Quantitative and qualitative experimental results demonstrate the efficacy of our framework in refining the semantic segmentation results with user interactions. The source code will be publicly available.

Figures (6)

• Figure 1: Existing customized approaches depend on customized networks for object segmentation, which are decoupled with off-the-shelf networks and require offline training with tailored supervision. Our on-the-fly approach builds upon off-the-shelf semantic segmentation networks and only works at the test-time.
• Figure 2: Overview of the InterPCSeg. The pipeline is divided into several steps: (1) Warm up the off-the-shelf semantic segmentation network; (2) Infer with the warmed network; (3) User assesses the current segmentation result and provide corrective clicks or complete the whole process; (4) Calculate the test-time loss based on user clicks and the current segmentation result; (5) Optimize the network parameters; (6) Refine the segmentation result by re-inference.
• Figure 3: Illustration of our proposed interaction simulation scheme. The simulated interactions are achieved in the following steps: (1) Calculate the error map by subtracting the segmentation output from the ground truth; (2) Cluster the error regions to obtain the obvious error regions; (3) Point density estimation on obvious error regions to omit the boundary points; (4) Sample and label the points of interest based on their density.
• Figure 4: The mIoU curves with respect to the number of clicks (NoC).
• Figure 5: An interactive segmentation process by our proposed framework. The top row represents the input point cloud and the error map of each segmentation result. The bottom row consists of the ground truth result, initial segmentation result, and the refined results. The corrective clicks (totally 15 clicks) are progressively provided, marked as colored dots on the initial and refined segmentation results.