FreqPDE: Rethinking Positional Depth Embedding for Multi-View 3D Object Detection Transformers

TL;DR

This work tackles the problem of 3D object detection from multi-view 2D images by addressing depth supervision weaknesses that hinder depth accuracy, boundary precision, and cross-view consistency. It introduces FreqPDE, a frequency-aware positional depth embedding framework built from three modules: FSPE for frequency-enhanced multi-scale features, CSDP for cross-view, scale-invariant depth prediction, and PDE for depth-aware 3D feature encoding in the detector. The approach employs hybrid depth supervision that combines explicit metric depth from LiDAR with implicit distribution priors from pseudo-depth, enabling robust learning across distances. Experiments on nuScenes demonstrate state-of-the-art performance, with clear gains in NDS and mAP, especially for distant and small objects, validating the effectiveness of frequency-domain depth embedding in 3D detection transformers. The proposed components collectively improve depth quality, cross-view coherence, and 3D query decoding, offering practical impact for real-world autonomous driving systems. $D_{i,j} = \omega D^{\rm C}_{i,j} + (1-\omega) D^{\rm R}_{i,j}$ is used to fuse depth estimates, and $PE_j = \mathrm{Sine}(f_{\rm proj}(D_{i,j}, K_j, E_j))$ encodes depth-aware positional information for 3D queries.

Abstract

Detecting 3D objects accurately from multi-view 2D images is a challenging yet essential task in the field of autonomous driving. Current methods resort to integrating depth prediction to recover the spatial information for object query decoding, which necessitates explicit supervision from LiDAR points during the training phase. However, the predicted depth quality is still unsatisfactory such as depth discontinuity of object boundaries and indistinction of small objects, which are mainly caused by the sparse supervision of projected points and the use of high-level image features for depth prediction. Besides, cross-view consistency and scale invariance are also overlooked in previous methods. In this paper, we introduce Frequency-aware Positional Depth Embedding (FreqPDE) to equip 2D image features with spatial information for 3D detection transformer decoder, which can be obtained through three main modules. Specifically, the Frequency-aware Spatial Pyramid Encoder (FSPE) constructs a feature pyramid by combining high-frequency edge clues and low-frequency semantics from different levels respectively. Then the Cross-view Scale-invariant Depth Predictor (CSDP) estimates the pixel-level depth distribution with cross-view and efficient channel attention mechanism. Finally, the Positional Depth Encoder (PDE) combines the 2D image features and 3D position embeddings to generate the 3D depth-aware features for query decoding. Additionally, hybrid depth supervision is adopted for complementary depth learning from both metric and distribution aspects. Extensive experiments conducted on the nuScenes dataset demonstrate the effectiveness and superiority of our proposed method.

Figures (8)

• Figure 1: Illustration of motivation. (a) Input image with a fixed resolution. (b) Missing depth supervision of distant vehicles (red circle) with sparse projected LiDAR points. (c) Pseudo depth map complements implicit distribution priors additionally. (d) Coverage comparisons of projected points with different input resolution and downsampling factors.
• Figure 2: Overview of our proposed FreqPDE framework. FreqPDE first extracts multi-scale features from multi-view images using an off-the-shelf visual encoder. Then the FSPE module constructs a feature pyramid through combining high-frequency edge clues and low-frequency semantics from low-level and high-level image features respectively. The CSDP module estimates multi-scale depth maps with cross-view attention based on the frequency-enhanced features. Finally, the PDE module encodes both semantic features and geometric depth embedding to generate 3D depth-aware features for object query decoding. Hybrid depth supervision is introduced to facilitate complementary depth learning.
• Figure 3: Illustration of FSPE module. After encoding multi-scale features, low-frequency semantics are extracted from the high-level features through adaptive low-pass filters. Then the Discrete Wavelet Transform (DWT) is conducted on the low-level features to decompose four sub-components of different frequencies. The low-frequency component of low-level features is combined with the extracted high-level semantics for boundary enhancement.
• Figure 4: Detailed structure of CSDP module. (a) illustrates the process of visual feature re-weighting with efficient channel attention and cross-view attention from different levels to achieve cross-view consistency and scale invariance respectively. (b) describes the details of hierarchy depth prediction, where both categorical and regressed depth are predicted accordingly, which are fused with a hyper-parameter $\omega$ to generate the final depth map. (c) and (d) show the process of efficient channel attention and cross-view attention respectively.
• Figure 5: Feature visualization and intra-category similarity (IntraSim). The brighter color indicates a higher IntraSim for the car.