SAMPro3D: Locating SAM Prompts in 3D for Zero-Shot Instance Segmentation

TL;DR

SAMPro3D enables zero-shot 3D instance segmentation by locating SAM prompts directly in 3D space and projecting them to 2D frames to obtain consistent masks across views. It introduces a View-Guided Prompt Selection and a Surface-Based Prompt Consolidation strategy to ensure high-quality and comprehensive 3D segmentation without any training. The method achieves competitive or superior results to both open-set and fully supervised baselines and introduces a ScanNet200-Fine50 dataset to better evaluate fine-grained segmentation. This framework highlights the potential of SAM-like models for 3D understanding and suggests future directions for interactive prompts and integration with advanced 3D representations.

Abstract

We introduce SAMPro3D for zero-shot instance segmentation of 3D scenes. Given the 3D point cloud and multiple posed RGB-D frames of 3D scenes, our approach segments 3D instances by applying the pretrained Segment Anything Model (SAM) to 2D frames. Our key idea involves locating SAM prompts in 3D to align their projected pixel prompts across frames, ensuring the view consistency of SAM-predicted masks. Moreover, we suggest selecting prompts from the initial set guided by the information of SAM-predicted masks across all views, which enhances the overall performance. We further propose to consolidate different prompts if they are segmenting different surface parts of the same 3D instance, bringing a more comprehensive segmentation. Notably, our method does not require any additional training. Extensive experiments on diverse benchmarks show that our method achieves comparable or better performance compared to previous zero-shot or fully supervised approaches, and in many cases surpasses human annotations. Furthermore, since our fine-grained predictions often lack annotations in available datasets, we present ScanNet200-Fine50 test data which provides fine-grained annotations on 50 scenes from ScanNet200 dataset. The project page can be accessed at https://mutianxu.github.io/sampro3d/.

Figures (17)

• Figure 1: We introduce SAMPro3D for zero-shot instance segmentation of 3D scenes. Given the 3D point cloud and posed RGB-D frames of 3D scenes, our approach uses the Segment Anything Model (SAM) kirillov2023sam on RGB frames to segment 3D instances. Our method does not require additional training on domain-specific data. See \ref{['fig:qualitative_result']} for more impressive results.
• Figure 2: The comparison of our key idea and others. Our method (b) locates SAM prompts in 3D, which aligns pixel prompts across frames, bringing the frame consistency of prompts and their masks, and can handle newly emerged instances. Here we use random colors to visualize 3D results for instance discrimination, so there is no correlation between the colors assigned to 2D and 3D instances.
• Figure 3: An overview of our SAMPro3D, with a primary focus on “prompt”. Given 3D scene point clouds with posed RGB-D frames, we locate SAM kirillov2023sam prompts in input 3D scenes and project them onto 2D frames to obtain 2D segmentation masks. Later, the initial prompts and their masks are selected (\ref{['alg:prompt_filter']}) and consolidated (\ref{['fig:prompt_consolidation']}), leveraging both multi-view and surface information. Finally, we project all input points onto each segmented frame to obtain the 3D segmentation result.
• Figure 4: The illustration of the partial segmentation problem and our Surface-based Prompt Consolidation strategy.
• Figure 5: Qualitative comparison of our method, SAM3D sam3d, Mask3D mask3d and ScanNet200's annotations scannet200, across various scenes in ScanNet200, from holistic to focused view. Mask3D does not treat floor and wall as instances, resulting in the absence of these two labels in its results. Better view in zoom and color.