DN-DETR: Accelerate DETR Training by Introducing Query DeNoising

TL;DR

DN-DETR tackles DETR's slow convergence by stabilizing bipartite matching through a denoising training scheme that injects noised ground-truth boxes and labels into the decoder. The method adds a parallel denoising stream and an attention-mask design to prevent information leakage, enabling the model to reconstruct original boxes and labels without engaging in the unstable Hungarian matching during learning. Empirically, DN-DETR yields substantial speedups and performance gains across DETR-like models (including Deformable DETR and Anchor DETR) and even extends to Faster R-CNN and Mask2Former, with gains up to +1.9 AP in 12-epoch training and strong results under 1x training settings. The approach is lightweight, broadly applicable, and supported by extensive ablations, demonstrating that denoising training is a practical, generalizable technique for accelerating and improving object detection and segmentation models.

Abstract

We present in this paper a novel denoising training method to speedup DETR (DEtection TRansformer) training and offer a deepened understanding of the slow convergence issue of DETR-like methods. We show that the slow convergence results from the instability of bipartite graph matching which causes inconsistent optimization goals in early training stages. To address this issue, except for the Hungarian loss, our method additionally feeds ground-truth bounding boxes with noises into Transformer decoder and trains the model to reconstruct the original boxes, which effectively reduces the bipartite graph matching difficulty and leads to a faster convergence. Our method is universal and can be easily plugged into any DETR-like methods by adding dozens of lines of code to achieve a remarkable improvement. As a result, our DN-DETR results in a remarkable improvement ($+1.9$AP) under the same setting and achieves the best result (AP $43.4$ and $48.6$ with $12$ and $50$ epochs of training respectively) among DETR-like methods with ResNet-$50$ backbone. Compared with the baseline under the same setting, DN-DETR achieves comparable performance with $50\%$ training epochs. Code is available at \url{https://github.com/FengLi-ust/DN-DETR}.

Figures (8)

• Figure 1: Convergence curve between our model DN-Deformable-DETR built upon Deformable DETR with denoising training and previous models under ResNet-50 backbone.
• Figure 2: The $IS$ of DAB-DETR and DN-DETR during training. For each method, we train $12$ epoch on the same setting. We test the change of the Hungarian matching between each two epochs on the Validation set as the $IS$.
• Figure 3: A comparison of DAB-DETR and DN-DETR on anchor-target distance.
• Figure 4: (a)(b)Some examples of anchors and targets for DAB-DETR and DN-DETR, respectively. Each arrow starts from an anchor and points to a target. The color of each arrow shows its $l_1$ length and cooler colors denote shorter arrows.
• Figure 5: Comparison of the cross-attention part DAB-DETR and our DN-DETR (a)DAB-DETR directly uses dynamically updated anchor boxes to provide both a reference query point $(x,y)$ and a reference anchor size $(w,h)$ to improve the cross-attention computation. (b) DN-DETR specifies the decoder embeddings as label embeddings and adds an indicator to differentiate the denoising task and matching task.