OpenTrack3D: Towards Accurate and Generalizable Open-Vocabulary 3D Instance Segmentation

TL;DR

OpenTrack3D presents a mesh-free, open-vocabulary 3D instance segmentation framework that couples a visual–spatial tracker with 2D open-vocabulary cues and an MLLM-based classifier. By maintaining cross-view consistency and introducing optional geometry refinement, it achieves state-of-the-art results on diverse benchmarks (ScanNet200, Replica, ScanNet++, SceneFun3D) while generalizing to mesh-free scenes. Ablation studies confirm the benefits of combining DINO-based appearance with voxel-based spatial signals, the superiority of MLLM over CLIP for complex queries, and the value of 2D–3D consistency refinements. The work also provides practical insights into runtime efficiency and prompts for multi-modal open-vocabulary reasoning, signaling a path toward scalable 3D data annotation and robust 3D perception in unstructured environments.

Abstract

Generalizing open-vocabulary 3D instance segmentation (OV-3DIS) to diverse, unstructured, and mesh-free environments is crucial for robotics and AR/VR, yet remains a significant challenge. We attribute this to two key limitations of existing methods: (1) proposal generation relies on dataset-specific proposal networks or mesh-based superpoints, rendering them inapplicable in mesh-free scenarios and limiting generalization to novel scenes; and (2) the weak textual reasoning of CLIP-based classifiers, which struggle to recognize compositional and functional user queries. To address these issues, we introduce OpenTrack3D, a generalizable and accurate framework. Unlike methods that rely on pre-generated proposals, OpenTrack3D employs a novel visual-spatial tracker to construct cross-view consistent object proposals online. Given an RGB-D stream, our pipeline first leverages a 2D open-vocabulary segmenter to generate masks, which are lifted to 3D point clouds using depth. Mask-guided instance features are then extracted using DINO feature maps, and our tracker fuses visual and spatial cues to maintain instance consistency. The core pipeline is entirely mesh-free, yet we also provide an optional superpoints refinement module to further enhance performance when scene mesh is available. Finally, we replace CLIP with a multi-modal large language model (MLLM), significantly enhancing compositional reasoning for complex user queries. Extensive experiments on diverse benchmarks, including ScanNet200, Replica, ScanNet++, and SceneFun3D, demonstrate state-of-the-art performance and strong generalization capabilities.

Figures (5)

• Figure 1: Qualitative and quantitative results of our method. Left: Visualization of our results on the SceneFun3D dataset. As shown, our method accurately localizes the target objects even under complex query. Right: Comparison with prior methods that do not rely on supervised 3D masks, where our approach consistently outperforms them across all benchmarks.
• Figure 2: Overview of our Proposal Generation and Proposal Refinement stages. Proposal Generation employs a 2D open-vocabulary segmenter to generate object masks, which are lifted to 3D point clouds using depth maps and subsequently denoised. Mask-guided instance features are extracted from DINO feature maps, and our tracker fuses both visual and spatial cues to maintain instance consistency across frames. Proposal Refinement further improves the 3D instances through Consistency Refinement and Geometry Refinement, while the merging process is omitted for clearer visualization.
• Figure 3: Comparison of CLIP and MLLM on fine-grained, complex textual queries. CLIP receives cropped object regions, while MLLM processes the full image with the target highlighted by a red box. Green boxes indicate predicted instances for each method. MLLM better captures complex descriptions and produces more accurate predictions, whereas CLIP often fails.
• Figure 4: Comparative Visualization of Open-Vocabulary 3D Instance Segmentation on ScanNet200. Red masks denote query-correlated 3D instances. The comparative results demonstrate our method's clear advantage in addressing long-tail categories, showcasing improved segmentation accuracy and robustness when compared to baseline approaches.
• Figure 5: Qualitative Results on ScanNet++ (left) and SceneFun3D (right). We demonstrate instance segmentation results for various text queries. The visualizations demonstrate our method's capability to delineate objects with arbitrary geometries while maintaining recognition accuracy for specific textual descriptions.