DEYO: DETR with YOLO for End-to-End Object Detection

TL;DR

DEYO tackles the dependence of DETR-based detectors on ImageNet pretraining and the instability of early bipartite matching by introducing step-by-step training: first pretrain a classic one-to-many detector to initialize the backbone and neck, then freeze them and train a lightweight end-to-end decoder. It fuses a YOLO-based one-to-many branch with a pure convolutional encoder and a Transformer-based decoder, enabling real-time end-to-end detection without NMS or extra data. The approach yields state-of-the-art real-time performance on COCO and competitive results on dense crowds datasets like CrowdHuman, while substantially reducing training costs (two-stage VRAM requirements of 16GB and 8GB). This work demonstrates a practical path to high-accuracy, fast detectors on custom datasets, with potential for further backbone/neck specialization and scaling strategies tailored to DEYO.

Abstract

The training paradigm of DETRs is heavily contingent upon pre-training their backbone on the ImageNet dataset. However, the limited supervisory signals provided by the image classification task and one-to-one matching strategy result in an inadequately pre-trained neck for DETRs. Additionally, the instability of matching in the early stages of training engenders inconsistencies in the optimization objectives of DETRs. To address these issues, we have devised an innovative training methodology termed step-by-step training. Specifically, in the first stage of training, we employ a classic detector, pre-trained with a one-to-many matching strategy, to initialize the backbone and neck of the end-to-end detector. In the second stage of training, we froze the backbone and neck of the end-to-end detector, necessitating the training of the decoder from scratch. Through the application of step-by-step training, we have introduced the first real-time end-to-end object detection model that utilizes a purely convolutional structure encoder, DETR with YOLO (DEYO). Without reliance on any supplementary training data, DEYO surpasses all existing real-time object detectors in both speed and accuracy. Moreover, the comprehensive DEYO series can complete its second-phase training on the COCO dataset using a single 8GB RTX 4060 GPU, significantly reducing the training expenditure. Source code and pre-trained models are available at https://github.com/ouyanghaodong/DEYO.

Figures (6)

• Figure 1: DEYO has surpassed other real-time object detectors in speed and accuracy; all detectors were exclusively trained on the COCO dataset without any additional datasets.
• Figure 2: We eliminated the encoder usage and instead employed the multi-scale features {P3, P4, P5} provided by the neck. Following feature projection, these features were utilized as input for the encoder while simultaneously generating candidate bounding boxes and filtering them through the query selector. Subsequently, this information was passed into a decoder with an auxiliary prediction head, enabling iterative optimization for generating bounding boxes and scores.
• Figure 3: We decouple the generation of bounding boxes from embeddings, allowing for more effective compression of multiscale information emanating from the neck through enhanced feature projection.
• Figure 4: DEYO inherits a one-to-many branch pre-trained bounding box head, transitioning the learning strategy of the bounding box head from dense to sparse rather than training from scratch.
• Figure 5: Owing to the initial phase of DEYO being pre-trained on the COCO dataset for the task of object detection, it was afforded the capability to employ more robust data augmentation strategies during the second stage of training. Consequently, DEYO's adoption of the Mosaic data augmentation technique did not result in performance degradation, in contrast to the experience with DETRs.