Towards 3D Objectness Learning in an Open World

TL;DR

The paper addresses the challenge of learning 3D objectness in open-world settings by proposing OP3Det, a class-agnostic, prompt-free detector that leverages 2D semantic priors from the Segment Anything Model and a cross-modal mixture-of-experts to fuse point-cloud and image features. It formalizes open-world 3D objectness, introduces a 3D object discovery pipeline with multi-scale point sampling, and employs a dynamic MoE router to adaptively weigh uni-modal and cross-modal cues. Empirically, OP3Det achieves strong cross-category and cross-dataset generalization, showing substantial improvements over closed-world and open-vocabulary baselines across indoor and outdoor benchmarks, and it demonstrates robust ablation performance validating the design choices. The approach enables scalable, real-world 3D perception by discovering all salient objects without relying on fixed vocabularies or text prompts, with potential extensions to 2D detection tasks.

Abstract

Recent advancements in 3D object detection and novel category detection have made significant progress, yet research on learning generalized 3D objectness remains insufficient. In this paper, we delve into learning open-world 3D objectness, which focuses on detecting all objects in a 3D scene, including novel objects unseen during training. Traditional closed-set 3D detectors struggle to generalize to open-world scenarios, while directly incorporating 3D open-vocabulary models for open-world ability struggles with vocabulary expansion and semantic overlap. To achieve generalized 3D object discovery, We propose OP3Det, a class-agnostic Open-World Prompt-free 3D Detector to detect any objects within 3D scenes without relying on hand-crafted text prompts. We introduce the strong generalization and zero-shot capabilities of 2D foundation models, utilizing both 2D semantic priors and 3D geometric priors for class-agnostic proposals to broaden 3D object discovery. Then, by integrating complementary information from point cloud and RGB image in the cross-modal mixture of experts, OP3Det dynamically routes uni-modal and multi-modal features to learn generalized 3D objectness. Extensive experiments demonstrate the extraordinary performance of OP3Det, which significantly surpasses existing open-world 3D detectors by up to 16.0% in AR and achieves a 13.5% improvement compared to closed-world 3D detectors.

Figures (7)

• Figure 1: Illustration for Prompt-free 3D objectness learning. (a) Closed-world detectors can only recognize pre-defined categories in the training dataset (yellow boxes). Although some 3D detectors can detect novel classes via pre-defined prompts (green boxes), they still cannot detect "all" when the given vocabularies are limited. (b) In comparison, our OP3Det can detect rare categories (red boxes) and better discover 3D objects without the requirement of any semantic labels and text prompts.
• Figure 2: The overview of OP3Det. We apply SAM to introduce abundant 2D semantic knowledge for 3D object discovery. Multi-scale point sampling is utilized in this process. The cross-modal MoE is then employed to guide data pathways for uni-modal and multi-modal features, allowing the model to dynamically adapt its reliance on unimodal or cross-modal information according to the scenario.
• Figure 3: Visualization of point sampling trategy. The segmentation masks from SAM contain many small fragments and object parts. By using multi-scale point sampling, these noisy masks can be mitigated. Post-processing with a 2D class-agnostic detector further improves the quality.
• Figure 4: The visualized results of OP3Det on the SUN RGB-D (the first row) and ScanNet (the second row) dataset. The red boxes are base classes and blue boxes are novel classes.
• Figure 5: Qualitative comparison with SAM3D yang2023sam3d. Benefiting from our contributions, our method OP3Det yields more accurate localization and precise discovery of novel objects.