Any3DIS: Class-Agnostic 3D Instance Segmentation by 2D Mask Tracking

TL;DR

A 3D-Aware 2D Mask Tracking module that uses robust 3D priors from a 2D mask segmentation and tracking foundation model (SAM-2) to ensure consistent object masks across video frames and leverages a dynamic programming algorithm to select an optimal set of views.

Abstract

Existing 3D instance segmentation methods frequently encounter issues with over-segmentation, leading to redundant and inaccurate 3D proposals that complicate downstream tasks. This challenge arises from their unsupervised merging approach, where dense 2D instance masks are lifted across frames into point clouds to form 3D candidate proposals without direct supervision. These candidates are then hierarchically merged based on heuristic criteria, often resulting in numerous redundant segments that fail to combine into precise 3D proposals. To overcome these limitations, we propose a 3D-Aware 2D Mask Tracking module that uses robust 3D priors from a 2D mask segmentation and tracking foundation model (SAM-2) to ensure consistent object masks across video frames. Rather than merging all visible superpoints across views to create a 3D mask, our 3D Mask Optimization module leverages a dynamic programming algorithm to select an optimal set of views, refining the superpoints to produce a final 3D proposal for each object. Our approach achieves comprehensive object coverage within the scene while reducing unnecessary proposals, which could otherwise impair downstream applications. Evaluations on ScanNet200 and ScanNet++ confirm the effectiveness of our method, with improvements across Class-Agnostic, Open-Vocabulary, and Open-Ended 3D Instance Segmentation tasks.

Figures (4)

• Figure 1: Comparison of our proposed approach, Any3DIS, with existing 3D instance segmentation methods such as Open3DIS nguyen2023open3dis. Open3DIS frequently encounters over-segmentation issues, generating redundant 3D proposals due to its unsupervised merging process. In contrast, our approach leverages robust guidance from 2D mask tracking to maintain consistent object segmentation across video frames, effectively reducing redundant proposals and enhancing segmentation accuracy.
• Figure 2: Difference between Open3DIS nguyen2023open3dis and our Any3DIS. Open3DIS segments all 2D masks across views first, then lifts them to 3D proposals for merging, whereas our approach tracks all 2D masks across views, lifts them directly to 3D proposals, and then refines these proposals through optimization.
• Figure 3: Overview of Any3DIS. We propose a novel class-agnostic approach for 3D instance segmentation through 2D mask tracking. Specifically, we identify each selected superpoint's "pivot view" within the RGB-D frame sequence, where it is most visible. Using the SAM 2 model ravi2024sam, we then obtain the 2D segmentation of this superpoint in the pivot view and track this 2D segmentation across other views. For each 2D mask obtained, we generate a 3D mask candidate by aggregating all 3D superpoints whose projections intersect with any of the 2D masks. These superpoints are then subjected to mask optimization to produce the final refined 3D mask proposal, which is added to the class-agnostic 3D instance segmentation bank, ready for downstream tasks.
• Figure 4: Qualitative Result on ScanNet++ Validation Set.: From left to right we show the input point cloud of 3D scenes, GT segmentation, Open3DIS nguyen2023open3dis, and Any3DIS (ours) results. Our approach achieves more accurate and consistent segmentation than Open3DIS.