Learning Geometry-Guided Depth via Projective Modeling for Monocular 3D Object Detection

TL;DR

This work tackles monocular 3D object detection by explicitly tying depth to multiple 2D/3D geometry elements through a principled projective formulation. It integrates a geometry-guided depth module into an end-to-end network, enabling geometry-aware representation learning that directly refines depth without extra post-processing. The authors introduce a holistic geometric formula, implement it as a network module, and address 2D-3D misalignment with a robust baseline; on KITTI, the method achieves state-of-the-art monocular results with about a $2.80 ext{p.}$ AP$_{40}$ gain on the moderate setting and real-time performance. This approach improves depth estimation quality and 3D localization by leveraging pose, dimensions, and projection geometry, offering practical implications for autonomous driving systems.

Abstract

As a crucial task of autonomous driving, 3D object detection has made great progress in recent years. However, monocular 3D object detection remains a challenging problem due to the unsatisfactory performance in depth estimation. Most existing monocular methods typically directly regress the scene depth while ignoring important relationships between the depth and various geometric elements (e.g. bounding box sizes, 3D object dimensions, and object poses). In this paper, we propose to learn geometry-guided depth estimation with projective modeling to advance monocular 3D object detection. Specifically, a principled geometry formula with projective modeling of 2D and 3D depth predictions in the monocular 3D object detection network is devised. We further implement and embed the proposed formula to enable geometry-aware deep representation learning, allowing effective 2D and 3D interactions for boosting the depth estimation. Moreover, we provide a strong baseline through addressing substantial misalignment between 2D annotation and projected boxes to ensure robust learning with the proposed geometric formula. Experiments on the KITTI dataset show that our method remarkably improves the detection performance of the state-of-the-art monocular-based method without extra data by 2.80% on the moderate test setting. The model and code will be released at https://github.com/YinminZhang/MonoGeo.

Figures (11)

• Figure 1: Visualization of the depth difference in geometry projection. Object depth critically relates to both the pose and position of the object. For instance, for two cars with the same height in both the 2D bounding box (in blue) and the 3D bounding box (in orange), the depth values of their centers differ by more than $5$ meters because of their distinct poses and positions.
• Figure 2: An overview of our proposed holistic geometric representation learning approach. We leverage a network to extract features from the monocular image. Then base detection branch is used for generating 2D/3D predictions with depth excluded from image features. The 2D/3D predictions are utilized by the holistic geometric representation learning branch to generate geometric features via the proposed holistic geometric formula implemented in a network module. The geometric features are concatenated with the image features from the backbone for depth estimation. Based on the depth and other 3D predictions from the base detection branch, the detectors outputs the 3D object detection results. The symbol ⓒ indicates a concatenation operation.
• Figure 3: Illustration of the detailed computing flow of the proposed holistic geometric formula as shown in Fig. \ref{['fig: framework']}. Particularly, $\Delta z_{\max}$ involves the 3D pose of objects to represent the maximum difference between the eight corners of the objects in the z-axis, and $\beta$ represents the angle between the bottom center of the object and the horizontal plane.
• Figure 4: Visualization of notations in different object observation angles: (a) $\theta$ in Bird’s Eye View, and (b) $\beta$ in right-side view.
• Figure 5: Qualitative results of our method for multi-class 3D object detection. We use orange box for cars, purple box for pedestrians, and green box for cyclists. All illustrated images are from the KITTI test set. Zoom in the image for more details.