ImOV3D: Learning Open-Vocabulary Point Clouds 3D Object Detection from Only 2D Images

TL;DR

This paper proposes a novel framework ImOV3D to leverage pseudo multimodal representation containing both images and point clouds (PC) to close the modality gap and shows that ImOV3D significantly outperforms existing methods, even in the absence of ground truth 3D training data.

Abstract

Open-vocabulary 3D object detection (OV-3Det) aims to generalize beyond the limited number of base categories labeled during the training phase. The biggest bottleneck is the scarcity of annotated 3D data, whereas 2D image datasets are abundant and richly annotated. Consequently, it is intuitive to leverage the wealth of annotations in 2D images to alleviate the inherent data scarcity in OV-3Det. In this paper, we push the task setup to its limits by exploring the potential of using solely 2D images to learn OV-3Det. The major challenges for this setup is the modality gap between training images and testing point clouds, which prevents effective integration of 2D knowledge into OV-3Det. To address this challenge, we propose a novel framework ImOV3D to leverage pseudo multimodal representation containing both images and point clouds (PC) to close the modality gap. The key of ImOV3D lies in flexible modality conversion where 2D images can be lifted into 3D using monocular depth estimation and can also be derived from 3D scenes through rendering. This allows unifying both training images and testing point clouds into a common image-PC representation, encompassing a wealth of 2D semantic information and also incorporating the depth and structural characteristics of 3D spatial data. We carefully conduct such conversion to minimize the domain gap between training and test cases. Extensive experiments on two benchmark datasets, SUNRGBD and ScanNet, show that ImOV3D significantly outperforms existing methods, even in the absence of ground truth 3D training data. With the inclusion of a minimal amount of real 3D data for fine-tuning, the performance also significantly surpasses previous state-of-the-art. Codes and pre-trained models are released on the https://github.com/yangtiming/ImOV3D.

Figures (7)

• Figure 1: Left: Traditional methods require paired RGB-D data for training and use single-modality point clouds as input during inference. Right: ImOV3D involves using a vast amount of 2D images to generate pseudo point clouds during the training phase, which are then rendered back into images. In the inference phase, with only point clouds as input, we still construct a pseudo-multimodal representation to enhance detection performance.
• Figure 2: Overview of ImOV3D: Our model takes 2D images as input and puts them into the Pseudo 3D Annotation Generator to produce pseudo annotations. These 2D images are also fed into the Point Cloud Lifting Module to generate pseudo point clouds. Subsequently, using the Point Cloud Renderer, these pseudo point clouds are rendered into pseudo images, which then get processed by a 2D open vocabulary detector to detect 2D proposals and transfer the 2D semantic information to 3D space. Armed with pseudo point clouds, annotations, and pseudo images data, we proceed to train a multimodal 3D detector.
• Figure 3: Illustration of 3D Data Revision Module: (a) The rotation correction module involves processing an RGB image through a Normal Estimator to generate a normal map. This map then helps extract a horizontal surface mask for identifying horizontal point clouds, from which normal vectors $N_{pred}$ are obtained. These vectors are aligned with the Z-axis to compute the rotation matrix $R$. (b) In the 3D box filtering module, prompts related to object dimensions are first provided to GPT-4 to determine the mean size for each category. This mean size is then used to filter out boxes that do not meet the threshold criteria.
• Figure 4: The results from different types of 2D rendering images include depth maps and pseudo images.
• Figure 4: Qualitative results include (a) 2D RGB images, (b) 2D depth maps with 2D OVDetector annotations, and (c) pseudo images with annotations from a fine-tuned 2D detector.