DQ3D: Depth-guided Query for Transformer-Based 3D Object Detection in Traffic Scenarios

TL;DR

DQ3D tackles the mislocalization and occlusion challenges in transformer-based multi-view 3D object detection by introducing a depth-guided query generator that samples reference points on or inside object surfaces using depth maps and 2D detections. It further leverages temporal query alignment to incorporate historical detections and a hybrid attention layer to fuse depth-guided and temporal queries in the decoder, reducing reliance on dense queries. On nuScenes, DQ3D achieves notable improvements over the StreamPETR baseline, with a reported $mAP$ increase of $6.3\%$ and $NDS$ increase of $4.3\%$, demonstrating stronger accuracy and robustness in both single-frame and multi-frame settings. The approach offers practical impact by delivering more reliable 3D detections in traffic scenarios while maintaining computational efficiency through targeted query placement and memory-efficient temporal fusion.

Abstract

3D object detection from multi-view images in traffic scenarios has garnered significant attention in recent years. Many existing approaches rely on object queries that are generated from 3D reference points to localize objects. However, a limitation of these methods is that some reference points are often far from the target object, which can lead to false positive detections. In this paper, we propose a depth-guided query generator for 3D object detection (DQ3D) that leverages depth information and 2D detections to ensure that reference points are sampled from the surface or interior of the object. Furthermore, to address partially occluded objects in current frame, we introduce a hybrid attention mechanism that fuses historical detection results with depth-guided queries, thereby forming hybrid queries. Evaluation on the nuScenes dataset demonstrates that our method outperforms the baseline by 6.3\% in terms of mean Average Precision (mAP) and 4.3\% in the NuScenes Detection Score (NDS).

Figures (9)

• Figure 1: Wrong detection by StreamPETR. Yellow 3D box represents a detected vehicle, which is actually an image in the mirror
• Figure 2: An illustration of (a) the Fixed query and (b) the proposed Depth-guided query. The Fixed query is generated by the reference points distributed throughout 3D space, which is coarse-grained and can be distant from the object in some cases. In contrast, the depth-guided query provides more accurate positional information by encoding the location of sampled points on an object's surface or within its interior, using an estimated depth.
• Figure 3: The proposed DQ3D framework. We introduce two key innovations: the Depth-guided Query Generator (DQG) that initializes depth-guided queries using depth maps and 2D detection boxes, and a hybrid attention layer in the decoder to fuse depth-guided and temporal queries for enhanced 3D object detection.
• Figure 4: Depth-guided query generator(DQG). DQG sample the 3D reference points using 2D detection box and depth map $D_{pred}$. 3D reference points are not only sampled on the surface of objects, but also deeper within the object based on the interval $\Delta D$. The position embedding of the query is computed based on the 3D coordinates of the reference point. The query's semantic embedding is sampled from image feature according to the 2D reference point position.
• Figure 5: Temporal query alignment(TQA). The center point$P_{ref}^{t-1}$ and semantic embedding $Q_{sem}^{t-1}$ of historical detection result is saved. TQA align these information with current frame by coordinate transform and MLP considering the ego motion and the object motion.