Weakly Supervised 3D Object Detection with Multi-Stage Generalization

TL;DR

The paper tackles the problem of monocular 3D object detection with only 2D annotations, addressing the data bottleneck by leveraging Structure-from-Motion to reconstruct scene-level geometry and derive object-level pseudo-labels. It introduces BA2-Det, a two-phase framework combining pseudo-label generation via DoubleClustering with a three-stage generalization (complete-to-partial, static-to-dynamic, near-to-far) and self-training to train a monocular detector. Key contributions include a novel clustering-based pseudo-label extraction, a dedicated network (G_theta) for partial-to-full box generalization, a 2D-guided label assignment and orientation loss to handle unlabeled moving objects, and temporal/object-centric refinement via OTCL and OBA. Experiments on Waymo Open Dataset and KITTI demonstrate competitive performance relative to fully supervised baselines and reveal strong pretraining gains, highlighting the practical potential for open-set 3D detection and scalable data-efficient learning, albeit with reconstruction-dependent limitations.

Abstract

With the rapid development of large models, the need for data has become increasingly crucial. Especially in 3D object detection, costly manual annotations have hindered further advancements. To reduce the burden of annotation, we study the problem of achieving 3D object detection solely based on 2D annotations. Thanks to advanced 3D reconstruction techniques, it is now feasible to reconstruct the overall static 3D scene. However, extracting precise object-level annotations from the entire scene and generalizing these limited annotations to the entire scene remain challenges. In this paper, we introduce a novel paradigm called BA$^2$-Det, encompassing pseudo label generation and multi-stage generalization. We devise the DoubleClustering algorithm to obtain object clusters from reconstructed scene-level points, and further enhance the model's detection capabilities by developing three stages of generalization: progressing from complete to partial, static to dynamic, and close to distant. Experiments conducted on the large-scale Waymo Open Dataset show that the performance of BA$^2$-Det is on par with the fully-supervised methods using 10% annotations. Additionally, using large raw videos for pretraining,BA$^2$-Det can achieve a 20% relative improvement on the KITTI dataset. The method also has great potential for detecting open-set 3D objects in complex scenes. Project page: https://ba2det.site.

Figures (7)

• Figure 1: Pipeline of BA$^2$-Det. Top: reconstruction-based pseudo label generation process. We cluster the object point clouds from the reconstructed scene and fit the tight bounding box as the pseudo label. Bottom: Three stages of network generalization. The neural networks inside the red rounded rectangles are also for the inference.
• Figure 2: Illustrations of fitted bounding boxes and generalized boxes. Some occluded objects are badly reconstructed, leading to inaccurate pseudo-labels. $G_\theta$ can generalize from augmented complete objects to partial objects.
• Figure 3: Comparison between 2D and 3D label assignment. For 3D label assignment used in fully supervised setting, moving objects are negative samples. However, our 2D label assignment keeps them as positive samples.
• Figure 4: Depth distributions of ground truth and pseudo labels. BA$^2$-Det can generate pseudo labels up to a maximum of 200m, whereas the ground truth labels range from 0m to 75m.
• Figure 5: Open-set 3D object detection from a video sequence. For the video demos for 3D box generation, please refer to our project page.