Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries

Yue Wang, Vitor Guizilini, Tianyuan Zhang, Yilun Wang, Hang Zhao, Justin Solomon

TL;DR

DETR3D tackles multi-view 3D object detection from RGB without relying on dense depth prediction or post-processing like NMS. It introduces a top-down, 3D-to-2D query framework that back-projects sparse 3D references into multi-view features and refines predictions iteratively via a set-based loss. Key contributions include a geometry-aware detection head that fuses information across cameras within each layer and elimination of NMS, achieving state-of-the-art results on nuScenes, especially in camera overlap regions. The approach demonstrates robustness to depth estimation errors and offers a fast, end-to-end alternative to traditional depth-based or per-view pipelines.

Abstract

We introduce a framework for multi-camera 3D object detection. In contrast to existing works, which estimate 3D bounding boxes directly from monocular images or use depth prediction networks to generate input for 3D object detection from 2D information, our method manipulates predictions directly in 3D space. Our architecture extracts 2D features from multiple camera images and then uses a sparse set of 3D object queries to index into these 2D features, linking 3D positions to multi-view images using camera transformation matrices. Finally, our model makes a bounding box prediction per object query, using a set-to-set loss to measure the discrepancy between the ground-truth and the prediction. This top-down approach outperforms its bottom-up counterpart in which object bounding box prediction follows per-pixel depth estimation, since it does not suffer from the compounding error introduced by a depth prediction model. Moreover, our method does not require post-processing such as non-maximum suppression, dramatically improving inference speed. We achieve state-of-the-art performance on the nuScenes autonomous driving benchmark.

DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries

TL;DR

DETR3D tackles multi-view 3D object detection from RGB without relying on dense depth prediction or post-processing like NMS. It introduces a top-down, 3D-to-2D query framework that back-projects sparse 3D references into multi-view features and refines predictions iteratively via a set-based loss. Key contributions include a geometry-aware detection head that fuses information across cameras within each layer and elimination of NMS, achieving state-of-the-art results on nuScenes, especially in camera overlap regions. The approach demonstrates robustness to depth estimation errors and offers a fast, end-to-end alternative to traditional depth-based or per-view pipelines.

Abstract

We introduce a framework for multi-camera 3D object detection. In contrast to existing works, which estimate 3D bounding boxes directly from monocular images or use depth prediction networks to generate input for 3D object detection from 2D information, our method manipulates predictions directly in 3D space. Our architecture extracts 2D features from multiple camera images and then uses a sparse set of 3D object queries to index into these 2D features, linking 3D positions to multi-view images using camera transformation matrices. Finally, our model makes a bounding box prediction per object query, using a set-to-set loss to measure the discrepancy between the ground-truth and the prediction. This top-down approach outperforms its bottom-up counterpart in which object bounding box prediction follows per-pixel depth estimation, since it does not suffer from the compounding error introduced by a depth prediction model. Moreover, our method does not require post-processing such as non-maximum suppression, dramatically improving inference speed. We achieve state-of-the-art performance on the nuScenes autonomous driving benchmark.

Paper Structure

This paper contains 15 sections, 5 equations, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multi-view 3D Object Detection
  4. Overview
  5. Feature Learning
  6. Detection Head
  7. Loss
  8. Experiments
  9. Implementation Details
  10. Implementation Details
  11. Comparison to Existing Works
  12. Comparison in Overlap Regions
  13. Comparison to pseudo-LiDAR Methods
  14. Ablation & Analysis
  15. Conclusion

Figures (3)

  • Figure 1: Overview of our method. The inputs to the model are a set of multi-view images, which are encoded by a ResNet and a FPN. Then, our model operates on a set of sparse object queries in which each query is decoded to a 3D reference point. 2D features are transformed to refine the object queries by projecting the 3D reference point into the image space. Our model makes per-query predictions and uses a set-to-set loss.
  • Figure 2: Detection results from layer 1 to layer 5 in the DETR3D head. We visualize the bounding boxes in the BEV and overlay the point clouds from lidar_top. The predictions get closer to the ground-truth in the deeper layers.
  • Figure 3: We visualize DETR3D predictions in both BEV and image views. Our model is capable of detecting rather small objects and even objects that were not annotated as ground-truth (cars in CAM_BACK_LEFT). Some failure cases include the far ahead car in CAM_FRONT, that was not detected.