MonoDETR: Depth-guided Transformer for Monocular 3D Object Detection

TL;DR

<3-5 sentence high-level summary> MonoDETR presents a DETR-inspired framework for monocular 3D object detection that is guided by depth cues rather than relying on local center features. It introduces a depth predictor, a depth encoder, and a depth-guided decoder with a depth cross-attention mechanism and depth positional encodings, enabling non-local interactions and scene-level depth reasoning. The model achieves state-of-the-art results on KITTI without dense depth annotations and demonstrates plug-and-play applicability to multi-view detectors like PETR v2 and BEVFormer, improving BEV and NDS metrics. This work provides a strong, end-to-end baseline for depth-aware monocular 3D detection and highlights the value of integrating explicit depth guidance into Transformer-based detectors.

Abstract

Monocular 3D object detection has long been a challenging task in autonomous driving. Most existing methods follow conventional 2D detectors to first localize object centers, and then predict 3D attributes by neighboring features. However, only using local visual features is insufficient to understand the scene-level 3D spatial structures and ignores the long-range inter-object depth relations. In this paper, we introduce the first DETR framework for Monocular DEtection with a depth-guided TRansformer, named MonoDETR. We modify the vanilla transformer to be depth-aware and guide the whole detection process by contextual depth cues. Specifically, concurrent to the visual encoder that captures object appearances, we introduce to predict a foreground depth map, and specialize a depth encoder to extract non-local depth embeddings. Then, we formulate 3D object candidates as learnable queries and propose a depth-guided decoder to conduct object-scene depth interactions. In this way, each object query estimates its 3D attributes adaptively from the depth-guided regions on the image and is no longer constrained to local visual features. On KITTI benchmark with monocular images as input, MonoDETR achieves state-of-the-art performance and requires no extra dense depth annotations. Besides, our depth-guided modules can also be plug-and-play to enhance multi-view 3D object detectors on nuScenes dataset, demonstrating our superior generalization capacity. Code is available at https://github.com/ZrrSkywalker/MonoDETR.

Figures (6)

• Figure 1: Center-guided (Top) and Depth-guided Paradigms (Bottom) for monocular 3D object detection. Existing center-guided methods predict 3D attributes from local visual features around the centers, while our MonoDETR guides the detection by a predicted foreground depth map and adaptively aggregates features in global context. The lower right figure visualizes the attention map of the target query in the depth cross-attention layer.
• Figure 2: Comparison of DETR-based methods for camera-based 3D object detection. We utilize yellow, blue, green, and red to respectively denote the feature or prediction space related with 2D, depth, 3D, and BEV. Different from other methods, our MonoDETR leverages depth cues to guide 3D object detection from monocular images.
• Figure 3: The lightweight depth predictor. We utilize the depth predictor to predict the depth features and foreground depth map, which only contains discrete object-wise depth values.
• Figure 4: Overall pipeline of MonoDETR. We first acquire the visual and depth features of the input image and utilize two parallel encoders for non-local encoding. Then, we propose a depth-guided decoder to adaptively aggregate scene-level features in global context.
• Figure 5: Plug-and-play for multi-view 3D object detection. We utilize yellow, blue, green, and red to respectively denote the feature space related to 2D, depth, 3D, and BEV. The depth-guided transformer of MonoDETR is adopted to enhance PETR (v2) liu2022petrliu2022petrv2 and BEVFormer li2022bevformer in a plug-and-play manner, which provides depth guidance from surrounding scenes.