V-MIND: Building Versatile Monocular Indoor 3D Detector with Diverse 2D Annotations

TL;DR

V-MIND tackles the scarcity of 3D training data for indoor monocular 3D detection by lifting large-scale 2D datasets into 3D space using a monocular depth estimator and camera intrinsics. It introduces a monocular 3D detector based on Cube R-CNN that classifies proposals in a vision-language space (CLIP embeddings) and predicts 3D boxes in a virtual depth space, enabling open-vocabulary capabilities. Two novel losses are proposed: a 3D self-calibration loss to mitigate lifting errors in pseudo boxes and an ambiguity loss to handle unlabeled new classes when augmenting with 2D data. Joint training with real 3D data and lifted pseudo-3D data yields state-of-the-art results on the Omni3D indoor dataset, with notable gains on new classes and improved long-tail recognition. This approach demonstrates the practical potential of leveraging abundant 2D annotations to scale monocular 3D perception without proportional increases in 3D labeling effort.

Abstract

The field of indoor monocular 3D object detection is gaining significant attention, fueled by the increasing demand in VR/AR and robotic applications. However, its advancement is impeded by the limited availability and diversity of 3D training data, owing to the labor-intensive nature of 3D data collection and annotation processes. In this paper, we present V-MIND (Versatile Monocular INdoor Detector), which enhances the performance of indoor 3D detectors across a diverse set of object classes by harnessing publicly available large-scale 2D datasets. By leveraging well-established monocular depth estimation techniques and camera intrinsic predictors, we can generate 3D training data by converting large-scale 2D images into 3D point clouds and subsequently deriving pseudo 3D bounding boxes. To mitigate distance errors inherent in the converted point clouds, we introduce a novel 3D self-calibration loss for refining the pseudo 3D bounding boxes during training. Additionally, we propose a novel ambiguity loss to address the ambiguity that arises when introducing new classes from 2D datasets. Finally, through joint training with existing 3D datasets and pseudo 3D bounding boxes derived from 2D datasets, V-MIND achieves state-of-the-art object detection performance across a wide range of classes on the Omni3D indoor dataset.

Figures (5)

• Figure 1: Augmented 2D-to-3D dataset generation. Given a 2D image dataset with diverse object classes, we first use a metric-based depth estimator to predict the depth map and an intrinsic predictor to derive the camera's intrinsic parameters. We then compute and denoise the object point cloud for each instance and generate a 3D bounding box (\ref{['method:3d-data-generation']}) that tightly encloses the point cloud.
• Figure 2: Overview of the proposed V-MIND. (a) Given an input image, a backbone model extracts features, which are then used to generate region proposals. Each proposal is classified in a pre-trained vision-language space, with a 3D bounding box predicted by the cube head (\ref{['method:3d-detector']}). Leveraging an augmented 2D-to-3D dataset, our model can detect not only the original object classes but also previously unseen ones, i.e., the new classes. (b) Since pseudo 3D boxes from the augmented dataset may contain lifting errors, we propose calibrating the predicted 3D boxes before computing the Chamfer loss. This adjustment helps prevent performance degradation caused by lifting errors (\ref{['method:self-calib']}). (c) In the 3D training dataset, objects from new classes may be treated as background during proposal assignment (\ref{['method:ambiguity_loss']}). To address this, we maximize the probability of the combined group of new classes and the background class, rather than just the background.
• Figure 3: Example to illustrate the ambiguity issue. Data from the 3D dataset with a limited vocabulary treats proposals enclosing new-class objects as background (e.g., proposal B in the figure), which interferes with the training of the 3D detector and hinders its accuracy in identifying new classes.
• Figure 4: Qualitative results of the 3D pseudo bounding boxes lifted from the LVISgupta2019lvis 2D dataset. The proposed 3D data generation method can produce accurate 3D bounding boxes in terms of categories, locations, and orientations.
• Figure 5: Qualitative results of V-MIND and baseline Cube R-CNN on (a) SUN-RGBD, (b) ARKitScenes, and (c) Hypersim. As demonstrated in (a), V-MIND effectively predicts new--class objects—specifically, a monitor and keyboard in the first row and a bag in the second row--while Cube R-CNN fails to do so. In (b), V-MIND successfully detects the partially occluded bathtub, whereas Cube R-CNN fails to do so. This demonstrates that incorporating diverse data from the augmented 2D-to-3D dataset has the potential to help address challenging detection scenarios, such as occlusion. For all examples, V-MIND shows the ability to detect a wider variety of object classes compared to the baseline Cube R-CNN.