Le-DETR: Revisiting Real-Time Detection Transformer with Efficient Encoder Design

TL;DR

It is demonstrated that with well-designed, real-time DETR models can achieve strong performance without the need for complex and computationally expensive pretraining.

Abstract

Real-time object detection is crucial for real-world applications as it requires high accuracy with low latency. While Detection Transformers (DETR) have demonstrated significant performance improvements, current real-time DETR models are challenging to reproduce from scratch due to excessive pre-training overheads on the backbone, constraining research advancements by hindering the exploration of novel backbone architectures. In this paper, we want to show that by using general good design, it is possible to have \textbf{high performance} with \textbf{low pre-training cost}. After a thorough study of the backbone architecture, we propose EfficientNAT at various scales, which incorporates modern efficient convolution and local attention mechanisms. Moreover, we re-design the hybrid encoder with local attention, significantly enhancing both performance and inference speed. Based on these advancements, we present Le-DETR (\textbf{L}ow-cost and \textbf{E}fficient \textbf{DE}tection \textbf{TR}ansformer), which achieves a new \textbf{SOTA} in real-time detection using only ImageNet1K and COCO2017 training datasets, saving about 80\% images in pre-training stage compared with previous methods. We demonstrate that with well-designed, real-time DETR models can achieve strong performance without the need for complex and computationally expensive pretraining. Extensive experiments show that Le-DETR-M/L/X achieves \textbf{52.9/54.3/55.1 mAP} on COCO Val2017 with \textbf{4.45/5.01/6.68 ms} on an RTX4090. It surpasses YOLOv12-L/X by \textbf{+0.6/-0.1 mAP} while achieving similar speed and \textbf{+20\%} speedup. Compared with DEIM-D-FINE, Le-DETR-M achieves \textbf{+0.2 mAP} with slightly faster inference, and surpasses DEIM-D-FINE-L by \textbf{+0.4 mAP} with only \textbf{0.4 ms} additional latency. Code and weights will be open-sourced.

Figures (5)

• Figure 2: Left: Comparison of Le-DETR and Real-Time DETR series in training overheads, While saving lots of training overheads offered by efficient encoder design, our model outperforms the previous SOTA model, D-FINE and DEIM-D-FINE.Right: Unlike previous works such as DEIM and D-FINE mainly focus on decoder architecture and training objects, we focus on efficient encoder design.
• Figure 3: The proposed Le-DETR is structured in three distinct stages: backbone, encoder, and decoder. Each first block of stages in this backbone serves as the downsampling block. In this figure, we also illustrate the encoder, which incorporates both the NAIFI and Feature Fusion components.
• Figure 4: Overview of EfficientNATBlock and Neighborhood Attention-based Improved Feature Inference(NAIFI).
• Figure 5: Visualization of applying Le-DETR-L into hard cases of object detection.
• Figure 6: Visualization of applying Le-DETR-X into hard cases of object detection.