Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Retrieving Objects from 3D Scenes with Box-Guided Open-Vocabulary Instance Segmentation

Khanh Nguyen, Dasith de Silva Edirimuni, Ghulam Mubashar Hassan, Ajmal Mian

TL;DR

This work tackles open-vocabulary 3D object retrieval in scene point clouds by enabling detection and segmentation of novel objects without heavy 2D segmentation models. It introduces a box-guided RGBD-based mask generation pipeline that uses a fast 2D open-vocabulary detector to propose 3D masks via superpoints and then classifies them through visibility-aware aggregation on 2D label maps. The method achieves faster inference and improved tail-class performance on ScanNet200 and Replica compared with SAM/CLIP-based OV-3DIS approaches, while maintaining strong 3D localization. Qualitative results demonstrate retrieval of rare objects like calendars, highlighting practical benefits for robotics and AR applications; the authors provide code to facilitate adoption. Overall, the approach offers a SAM/CLIP-free, efficient, and generalizable solution for open-vocabulary 3D object retrieval in real-world scenes.

Abstract

Locating and retrieving objects from scene-level point clouds is a challenging problem with broad applications in robotics and augmented reality. This task is commonly formulated as open-vocabulary 3D instance segmentation. Although recent methods demonstrate strong performance, they depend heavily on SAM and CLIP to generate and classify 3D instance masks from images accompanying the point cloud, leading to substantial computational overhead and slow processing that limit their deployment in real-world settings. Open-YOLO 3D alleviates this issue by using a real-time 2D detector to classify class-agnostic masks produced directly from the point cloud by a pretrained 3D segmenter, eliminating the need for SAM and CLIP and significantly reducing inference time. However, Open-YOLO 3D often fails to generalize to object categories that appear infrequently in the 3D training data. In this paper, we propose a method that generates 3D instance masks for novel objects from RGB images guided by a 2D open-vocabulary detector. Our approach inherits the 2D detector's ability to recognize novel objects while maintaining efficient classification, enabling fast and accurate retrieval of rare instances from open-ended text queries. Our code will be made available at https://github.com/ndkhanh360/BoxOVIS.

Retrieving Objects from 3D Scenes with Box-Guided Open-Vocabulary Instance Segmentation

TL;DR

This work tackles open-vocabulary 3D object retrieval in scene point clouds by enabling detection and segmentation of novel objects without heavy 2D segmentation models. It introduces a box-guided RGBD-based mask generation pipeline that uses a fast 2D open-vocabulary detector to propose 3D masks via superpoints and then classifies them through visibility-aware aggregation on 2D label maps. The method achieves faster inference and improved tail-class performance on ScanNet200 and Replica compared with SAM/CLIP-based OV-3DIS approaches, while maintaining strong 3D localization. Qualitative results demonstrate retrieval of rare objects like calendars, highlighting practical benefits for robotics and AR applications; the authors provide code to facilitate adoption. Overall, the approach offers a SAM/CLIP-free, efficient, and generalizable solution for open-vocabulary 3D object retrieval in real-world scenes.

Abstract

Locating and retrieving objects from scene-level point clouds is a challenging problem with broad applications in robotics and augmented reality. This task is commonly formulated as open-vocabulary 3D instance segmentation. Although recent methods demonstrate strong performance, they depend heavily on SAM and CLIP to generate and classify 3D instance masks from images accompanying the point cloud, leading to substantial computational overhead and slow processing that limit their deployment in real-world settings. Open-YOLO 3D alleviates this issue by using a real-time 2D detector to classify class-agnostic masks produced directly from the point cloud by a pretrained 3D segmenter, eliminating the need for SAM and CLIP and significantly reducing inference time. However, Open-YOLO 3D often fails to generalize to object categories that appear infrequently in the 3D training data. In this paper, we propose a method that generates 3D instance masks for novel objects from RGB images guided by a 2D open-vocabulary detector. Our approach inherits the 2D detector's ability to recognize novel objects while maintaining efficient classification, enabling fast and accurate retrieval of rare instances from open-ended text queries. Our code will be made available at https://github.com/ndkhanh360/BoxOVIS.

Paper Structure

This paper contains 22 sections, 2 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Open-Vocabulary 2D Scene Understanding
  4. Open-Vocabulary 3D Instance Segmentation (OV-3DIS).
  5. Method
  6. Box-Guided RGBD-Based Mask Generation
  7. Box-Guided RGBD-Based Mask Generation
  8. Box-Based Mask Classification
  9. Constructing Label Maps.
  10. Computing Visibility.
  11. Aggregating Class Distribution.
  12. Experiments
  13. Experimental Setup
  14. Datasets and Evaluation Metric.
  15. Implementation Details.
  16. ...and 7 more sections

Figures (2)

  • Figure 1: Overview of the proposed method.
  • Figure 2: Visualized predictions of Open-YOLO 3D and our method on scene scene0353_00 in ScanNet200 dataset. Open-YOLO 3D completely relies on the pretrained 3D network for instance proposals, unable to retrieve the "calendar" instance from the scene while our method can leverage the 2D detector to form extra masks to cover low-frequency classes.