Monocular 3D Object Detection with Depth from Motion

TL;DR

This work introduces Depth from Motion (DfM), a monocular 3D object detection framework that exploits ego-motion between frames to infer depth through a geometry-aware cost volume, augmented by a monocular understanding pathway. By formulating depth estimation as a learned plane-sweep problem in a canonical space and fusing monocular cues, DfM addresses the ill-posed nature of monocular depth and improves 3D detection performance on KITTI, even without pose data via a self-supervised pose-learning variant. Key contributions include a geometry-aware stereo cost volume, a learnable monocular compensation mechanism, and a pose-free depth-from-motion design with self-supervised training, all validated by extensive quantitative and qualitative analyses and ablations. The approach achieves state-of-the-art results among monocular methods on KITTI, demonstrates interpretable fusion weights, and provides practical flexibility for pose-free deployment and broader applications in camera-only 3D perception.

Abstract

Perceiving 3D objects from monocular inputs is crucial for robotic systems, given its economy compared to multi-sensor settings. It is notably difficult as a single image can not provide any clues for predicting absolute depth values. Motivated by binocular methods for 3D object detection, we take advantage of the strong geometry structure provided by camera ego-motion for accurate object depth estimation and detection. We first make a theoretical analysis on this general two-view case and notice two challenges: 1) Cumulative errors from multiple estimations that make the direct prediction intractable; 2) Inherent dilemmas caused by static cameras and matching ambiguity. Accordingly, we establish the stereo correspondence with a geometry-aware cost volume as the alternative for depth estimation and further compensate it with monocular understanding to address the second problem. Our framework, named Depth from Motion (DfM), then uses the established geometry to lift 2D image features to the 3D space and detects 3D objects thereon. We also present a pose-free DfM to make it usable when the camera pose is unavailable. Our framework outperforms state-of-the-art methods by a large margin on the KITTI benchmark. Detailed quantitative and qualitative analyses also validate our theoretical conclusions. The code will be released at https://github.com/Tai-Wang/Depth-from-Motion.

Figures (8)

• Figure 1: In this paper, we present a framework for monocular 3D detection from videos. In contrast to previous work only relying on monocular understanding from a single image, our method integrates the stereo geometric clues from temporally adjacent images. It significantly improves depth estimation accuracy, the most critical part for camera-only 3D perception, and thus enhances the 3D detection performance.
• Figure 2: Multi-view geometry for object depth estimation in the (a) binocular, (b) parallel two-view, (c) general two-view system and (d) that for moving objects.
• Figure 3: An overview of our framework.
• Figure 4: Key components in our depth-from-motion module.
• Figure 5: Qualitative Analysis of aggregation weights in different cases.