SEED: A Simple and Effective 3D DETR in Point Clouds

TL;DR

The paper tackles the challenge of applying DETR to 3D point clouds by identifying two main bottlenecks: query selection in sparse data and effective query interaction that exploits geometric structure. It proposes SEED, a simple yet effective 3D DETR head that combines Dual Query Selection (DQS) with Deformable Grid Attention (DGA), and augments it with Quality-aware Hungarian Matching (QHM) for ground-truth assignment. The design yields high-quality queries (coarse-to-fine via DQS) and geometry-aware feature interaction (DGA) while using QHM to better align predictions with ground truth; the approach achieves state-of-the-art results on Waymo and nuScenes among DETR-based methods, with competitive speed. Overall, SEED provides a strong, scalable baseline for 3D DETR in point clouds and highlights the value of targeted query selection and geometry-guided interaction for LiDAR-based detection.

Abstract

Recently, detection transformers (DETRs) have gradually taken a dominant position in 2D detection thanks to their elegant framework. However, DETR-based detectors for 3D point clouds are still difficult to achieve satisfactory performance. We argue that the main challenges are twofold: 1) How to obtain the appropriate object queries is challenging due to the high sparsity and uneven distribution of point clouds; 2) How to implement an effective query interaction by exploiting the rich geometric structure of point clouds is not fully explored. To this end, we propose a simple and effective 3D DETR method (SEED) for detecting 3D objects from point clouds, which involves a dual query selection (DQS) module and a deformable grid attention (DGA) module. More concretely, to obtain appropriate queries, DQS first ensures a high recall to retain a large number of queries by the predicted confidence scores and then further picks out high-quality queries according to the estimated quality scores. DGA uniformly divides each reference box into grids as the reference points and then utilizes the predicted offsets to achieve a flexible receptive field, allowing the network to focus on relevant regions and capture more informative features. Extensive ablation studies on DQS and DGA demonstrate its effectiveness. Furthermore, our SEED achieves state-of-the-art detection performance on both the large-scale Waymo and nuScenes datasets, illustrating the superiority of our proposed method. The code is available at https://github.com/happinesslz/SEED

Figures (9)

• Figure 1: Comparison with DETR-based detectors zhu2023conquerchen2023focalformer3dbai2022transfusion and other representative methods chen2023voxelnextshi2021pv on Waymo validation dataset sun2020scalability in terms of detection performance and running speed. For a fair comparison, we evaluate the running speed of all approaches on an NVIDIA GeForce RTX 3090 with a batch size of 1. -S, -B and -L means the small, base, large versions of our SEED, respectively.
• Figure 2: Overall architecture of SEED, which consists of a 3D backbone and a SEED detection head. Specifically, the proposed SEED detection head mainly includes a dual query selection (DQS) module and a transformer decoder. The DQS utilizes a coarse-to-fine query selection strategy to select high-quality queries. The transformer decoder, including six SEED decoder layers, takes these queries as inputs and then iteratively performs a self-attention operation for inter-query interaction and a proposed deformable grid attention (DGA) for feature interaction between query and BEV features, generating final detection results.
• Figure 3: Illustration of dual query selection (DQS). DQS adopts a coarse-to-fine manner, which consists of a foreground query selection and a quality query selection. $S_c$, $S_l$, and $B_c$ are the predicted classification score, localization score, and regression for proposal boxes through three feed-forward networks (FFN) branches, respectively.
• Figure 4: Illustration of deformable grid attention (DGA). DGA first uniformly divides each reference proposal into grids as the reference points and then utilizes the predicted offsets to achieve a flexible receptive field.
• Figure 5: Visualization of SEED without DQS (the first row) and with DQS (the second row). We highlight the challenging queries with red circles. The colormap indicates the values of the confidence scores for selected queries on the BEV map. Green boxes are the ground truth boxes.