DenseBEV: Transforming BEV Grid Cells into 3D Objects

TL;DR

DenseBEV tackles inefficiencies in BEV-based camera 3D detectors by replacing random anchors with a dense BEV grid as object priors. It introduces a two-stage BEV object query mechanism and BEV-NMS to prune duplicates, enabling end-to-end training without auxiliary detectors. A hybrid temporal modeling approach merges BEV-temporal features with a memory queue of past detections to enhance performance, especially for small objects. On nuScenes and Waymo, DenseBEV and DenseBEV++ achieve state-of-the-art results, notably improving small-object detection and achieving leading LET-mAP on Waymo, demonstrating strong practical impact for autonomous-driving perception.

Abstract

In current research, Bird's-Eye-View (BEV)-based transformers are increasingly utilized for multi-camera 3D object detection. Traditional models often employ random queries as anchors, optimizing them successively. Recent advancements complement or replace these random queries with detections from auxiliary networks. We propose a more intuitive and efficient approach by using BEV feature cells directly as anchors. This end-to-end approach leverages the dense grid of BEV queries, considering each cell as a potential object for the final detection task. As a result, we introduce a novel two-stage anchor generation method specifically designed for multi-camera 3D object detection. To address the scaling issues of attention with a large number of queries, we apply BEV-based Non-Maximum Suppression, allowing gradients to flow only through non-suppressed objects. This ensures efficient training without the need for post-processing. By using BEV features from encoders such as BEVFormer directly as object queries, temporal BEV information is inherently embedded. Building on the temporal BEV information already embedded in our object queries, we introduce a hybrid temporal modeling approach by integrating prior detections to further enhance detection performance. Evaluating our method on the nuScenes dataset shows consistent and significant improvements in NDS and mAP over the baseline, even with sparser BEV grids and therefore fewer initial anchors. It is particularly effective for small objects, enhancing pedestrian detection with a 3.8% mAP increase on nuScenes and an 8% increase in LET-mAP on Waymo. Applying our method, named DenseBEV, to the challenging Waymo Open dataset yields state-of-the-art performance, achieving a LET-mAP of 60.7%, surpassing the previous best by 5.4%. Code is available at https://github.com/mdaehl/DenseBEV.

Figures (8)

• Figure 1: Comparison of anchor generation methods for BEV-based transformers in multi-camera 3D object detection. Previous methods (see upper half) use randomly generated anchors Zhiqi2022bev or employ detection-based anchors that either replace jiang2024far3d or complement yang2023bevformer randomly generated anchors. Our method employs a BEV feature grid as anchors and refines them using NMS.
• Figure 2: Overall architecture of our proposed method, DenseBEV. Images are processed by a backbone network and encoded using the BEVFormer encoder Zhiqi2022bev. Instead of introducing additional object queries, DenseBEV utilizes the encoder output as object queries after passing them through the suppression block. To minimize duplicate predictions, a suppression block is placed after each decoder layer. The dashed line indicates that the gradient of the object queries is detached from the encoder.
• Figure 3: Visualization of attention mask. Initially, all object queries can attend to each other (blue squares), but after the decoding and NMS process, suppressed ones are marked as invalid (red squares) in the attention mask.
• Figure 4: Hybrid Temporal Modeling. Detections from the previous timestamp, shown in different colors in the object memory, are concatenated with current BEV queries to form extended object queries. A memory queue containing detections from multiple earlier timestamps is also incorporated into the keys and values of subsequent attention layers. For clarity, motion layer normalizations from StreamPETR wang2023exploring are omitted.
• Figure 5: On the left, a BEV grid section near the ego vehicle (black box with forward arrow) is shown. The red square highlights the anchor grid position used to detect the corresponding bounding box, linked by a red line. The right side shows the related camera images for context.