LabelAny3D: Label Any Object 3D in the Wild

TL;DR

LabelAny3D introduces an analysis-by-synthesis pipeline that reconstructs holistic 3D scenes from monocular images to generate high-quality 3D bounding boxes, enabling scalable annotation for open-world 3D detection. By integrating super-resolution, amodal completion, TRELLIS-based 3D reconstruction, depth alignment, and precise 2D–3D pose alignment with vision foundation models, it outputs metric-scale 3D boxes suitable for training detectors. The authors curate COCO3D, a diverse in-the-wild benchmark, and demonstrate that pseudo-labels produced by LabelAny3D improve open-vocabulary monocular 3D detection and generalization to novel categories, outperforming prior auto-labelers. This work highlights the potential of foundation-model–driven annotation to scale 3D recognition in realistic settings and lays groundwork for broader 3D scene understanding tasks.

Abstract

Detecting objects in 3D space from monocular input is crucial for applications ranging from robotics to scene understanding. Despite advanced performance in the indoor and autonomous driving domains, existing monocular 3D detection models struggle with in-the-wild images due to the lack of 3D in-the-wild datasets and the challenges of 3D annotation. We introduce LabelAny3D, an \emph{analysis-by-synthesis} framework that reconstructs holistic 3D scenes from 2D images to efficiently produce high-quality 3D bounding box annotations. Built on this pipeline, we present COCO3D, a new benchmark for open-vocabulary monocular 3D detection, derived from the MS-COCO dataset and covering a wide range of object categories absent from existing 3D datasets. Experiments show that annotations generated by LabelAny3D improve monocular 3D detection performance across multiple benchmarks, outperforming prior auto-labeling approaches in quality. These results demonstrate the promise of foundation-model-driven annotation for scaling up 3D recognition in realistic, open-world settings.

Figures (10)

• Figure 1: Samples from our proposed COCO3D benchmark.
• Figure 2: Overview.(a) Omni3D brazil2023omni3d offers large-scale 3D annotations but primarily covers indoor and self-driving scenarios. (b) The proposed LabelAny3D reconstructs 3D scenes (left) to annotate objects in 3D (right). (c) Leveraging LabelAny3D pseudo-labels to train a monocular 3D detector raises its $\text{AP}_{\text{3D}}$ (average precision; higher is better) on both Omni3D novel categories yao2024open and our new COCO3D benchmark.
• Figure 3: LabelAny3D. (a) Given an image, we first extract high-resolution object crops; (b) A holistic 3D scene is then built from robust depth estimation, 3D object reconstruction, and 2D-3D alignment algorithms. (c) Lastly, 3D labels can be easily extracted from the reconstructed 3D scene.
• Figure 4: COCO3D benchmark.(a) Distribution of the top 50 categories in the COCO3D benchmark. (b) Super-category-wise distribution in the COCO3D benchmark, based on MS-COCO mscoco. (c) Examples of samples removed from COCO3D during the human refinement process.
• Figure 5: Qualitative open-vocabulary 3D detection results on in-the-wild-images: OVMono3D yao2024openvs. our fine-tuned OVMono3D. We display both the 3D predictions overlaid on the image and a top-down view with a base grid of $1\,\text{m} \times 1\,\text{m}$ tiles.