Probabilistic Interactive 3D Segmentation with Hierarchical Neural Processes

TL;DR

NPISeg3D introduces a probabilistic, hierarchical Neural Process framework for interactive 3D segmentation that robustly generalizes from sparse user clicks and provides explicit uncertainty estimates. By embedding scene-level and object-level latent variables and a probabilistic prototype modulator, the method adapts to diverse objects and scenes, while Monte Carlo sampling yields informative uncertainty maps to guide user interaction. Empirical results across indoor and outdoor datasets show improved segmentation accuracy with fewer clicks, and reliable uncertainty quantification that benefits both multi- and single-object tasks, including challenging KITTI-360 scenarios. This approach advances practical interactive 3D segmentation by coupling few-shot generalization with interpretable uncertainty, enabling more efficient and reliable user interactions in real-world applications.

Abstract

Interactive 3D segmentation has emerged as a promising solution for generating accurate object masks in complex 3D scenes by incorporating user-provided clicks. However, two critical challenges remain underexplored: (1) effectively generalizing from sparse user clicks to produce accurate segmentation, and (2) quantifying predictive uncertainty to help users identify unreliable regions. In this work, we propose NPISeg3D, a novel probabilistic framework that builds upon Neural Processes (NPs) to address these challenges. Specifically, NPISeg3D introduces a hierarchical latent variable structure with scene-specific and object-specific latent variables to enhance few-shot generalization by capturing both global context and object-specific characteristics. Additionally, we design a probabilistic prototype modulator that adaptively modulates click prototypes with object-specific latent variables, improving the model's ability to capture object-aware context and quantify predictive uncertainty. Experiments on four 3D point cloud datasets demonstrate that NPISeg3D achieves superior segmentation performance with fewer clicks while providing reliable uncertainty estimations.

Figures (9)

• Figure 1: Overview of NPISeg3D. We formulate interactive 3D segmentation as a probabilistic modeling problem with neural processes. Given a 3D scene $S$ and a user click set $\mathcal{C}$, a point encoder encodes them into click prototypes $\mathbf{X}_C$ (context data) and scene features $\mathbf{X}_T$ (target data). Then, we introduce two hierarchical latent variables: scene-level latent variable $\mathbf{z}_s$ and object-level latent variable $\mathbf{z}_o$, to enable probabilistic modeling and capture contextual information across hierarchical levels. In probabilistic prototype modulator, each object-specific latent variable is utilized to generate object-specific weights $(\gamma, \beta)$, which modulate its corresponding click prototypes, thereby enhancing few-shot generalization and providing reliable uncertainty estimation. The posterior distributions of the latent variables are inferred from the target set $(\mathbf{X}_T,\mathbf{Y}_T)$, which supervise the prior during training.
• Figure 2: Computational Graph of our NPISeg3D. The framework incorporates a hierarchical inference structure with a scene-level latent variable ($z_s$) and an object-level latent variable ($z_o$), capturing contextual information at different spatial levels.
• Figure 3: Qualitative results on interactive multi-object segmentation on S3DIS and KITTI360. Newly added clicks are represented by dark-colored dots. Please zoom in for more details.
• Figure 4: Uncertainty maps of predictions with increasing clicks. We show mask predictions and uncertainty after $K$ clicks.
• Figure 5: Qualitative comparison between the state-of-the-art (SoTA) method AGILE3D and our proposed NPISeg3D on the interactive multi-object segmentation task. Newly added clicks are represented by dark-colored dots. Our NPISeg3D consistently achieves higher IoU scores with the same number of clicks, particularly on the challenging outdoor LiDAR dataset KITTI-360.