General Geometry-aware Weakly Supervised 3D Object Detection

TL;DR

The paper tackles the high cost of 3D annotation for object detection by introducing General Geometry-Aware (GGA), a framework that learns 3D detectors from RGB images and 2D boxes using LLM-derived priors and geometry-based constraints. It decomposes the problem into prior injection (SRL from LLMs), 2D space projection via Boundary Projection Loss, and 3D space geometry refinement through Points-to-Box Alignment Loss, enabling effective weakly supervised learning across outdoor and indoor domains. Key contributions include the SRL, BPL, and PAL modules, an In-Box-Points-based pseudo supervision scheme, and demonstrable generalization on KITTI and SUN-RGBD with strong gains over prior weakly supervised methods. The approach reduces annotation costs while delivering high-quality 3D boxes, with practical impact for scalable 3D scene understanding in autonomous driving and robotics.

Abstract

3D object detection is an indispensable component for scene understanding. However, the annotation of large-scale 3D datasets requires significant human effort. To tackle this problem, many methods adopt weakly supervised 3D object detection that estimates 3D boxes by leveraging 2D boxes and scene/class-specific priors. However, these approaches generally depend on sophisticated manual priors, which is hard to generalize to novel categories and scenes. In this paper, we are motivated to propose a general approach, which can be easily adapted to new scenes and/or classes. A unified framework is developed for learning 3D object detectors from RGB images and associated 2D boxes. In specific, we propose three general components: prior injection module to obtain general object geometric priors from LLM model, 2D space projection constraint to minimize the discrepancy between the boundaries of projected 3D boxes and their corresponding 2D boxes on the image plane, and 3D space geometry constraint to build a Point-to-Box alignment loss to further refine the pose of estimated 3D boxes. Experiments on KITTI and SUN-RGBD datasets demonstrate that our method yields surprisingly high-quality 3D bounding boxes with only 2D annotation. The source code is available at https://github.com/gwenzhang/GGA.

Figures (5)

• Figure 1: The insight of our proposed method. Given RGB-D data, we utilize semantic information, 2D space projection constraints, and 3D space geometry constraints to estimate 3D boxes.
• Figure 2: Illustration of our proposed GGA approach. Initially, point clouds of a given scene are processed through the point cloud feature learning backbone and proposal head. Meanwhile, we extract the In-Box-Points from the 3D frustum corresponding to a 2D bounding box. The prior ratio information is obtained from GPT-4 and incorporated into the network through SRL. The PAL provides the 3D space geometry constraint, while the BPL enforces the 2D space projection constraint.
• Figure 3: Generated pseudo 3D boxes of pedestrians by our proposed GGA.
• Figure 4: Visualization of generated 3D boxes for indoor scenarios.
• Figure 5: Visualization of generated 3D boxes in bird's eye view for outdoor scenarios.