RT-DETRv4: Painlessly Furthering Real-Time Object Detection with Vision Foundation Models

TL;DR

RT-DETRv4 tackles the semantic bottleneck in real-time DETR-based detectors by distilling high-level semantics from Vision Foundation Models into a training-time module. It introduces the Deep Semantic Injector (DSI) to align a high-level feature map $F_5$ with a VFM teacher via a lightweight Feature Projector and a cosine similarity alignment, and Gradient-guided Adaptive Modulation (GAM) to dynamically balance semantic supervision with the detection objective using gradient norms. Trained with a frozen DINOv3-ViT-B teacher, RT-DETRv4 achieves state-of-the-art COCO AP across model scales (AP up to $57.0$ at $78$ FPS for X) without any extra inference cost, outperforming prior real-time detectors. The approach is deployment-friendly and scalable to various VFMs and DETR-based detectors, offering a practical pathway to leverage foundation-model semantics for efficient real-time perception.

Abstract

Real-time object detection has achieved substantial progress through meticulously designed architectures and optimization strategies. However, the pursuit of high-speed inference via lightweight network designs often leads to degraded feature representation, which hinders further performance improvements and practical on-device deployment. In this paper, we propose a cost-effective and highly adaptable distillation framework that harnesses the rapidly evolving capabilities of Vision Foundation Models (VFMs) to enhance lightweight object detectors. Given the significant architectural and learning objective disparities between VFMs and resource-constrained detectors, achieving stable and task-aligned semantic transfer is challenging. To address this, on one hand, we introduce a Deep Semantic Injector (DSI) module that facilitates the integration of high-level representations from VFMs into the deep layers of the detector. On the other hand, we devise a Gradient-guided Adaptive Modulation (GAM) strategy, which dynamically adjusts the intensity of semantic transfer based on gradient norm ratios. Without increasing deployment and inference overhead, our approach painlessly delivers striking and consistent performance gains across diverse DETR-based models, underscoring its practical utility for real-time detection. Our new model family, RT-DETRv4, achieves state-of-the-art results on COCO, attaining AP scores of 49.7/53.5/55.4/57.0 at corresponding speeds of 273/169/124/78 FPS.

Figures (5)

• Figure 1: Compared with existing advanced real-time object detectors on COCO lin2014microsoft. Our RT-DETRv4 models achieve state-of-the-art performance.
• Figure 2: Overview of RT-DETRv4. We leverage a Vision Foundation Model (VFM) to extract high-quality semantic representations, which are aligned with the deepest feature map ($F_5$) from the AIFI module via a Feature Projector in the Deep Semantic Injector (DSI). To ensure faster and more stable convergence, a Gradient-guided Adaptive Modulation (GAM) dynamically adjusts the DSI loss during training. The proposed framework operates only during the training phase (highlighted by dashed arrows and blue blocks) of the real-time detector and keeps the original architecture unchanged during inference and deployment, introducing no additional overhead while improving accuracy.
• Figure 3: Illustration of different Deep Semantic Injector (DSI) strategies. (a) Direct alignment of multi-scale backbone features ($S_3, S_4, S_5$). (b) Hybrid alignment of both backbone features and the AIFI output feature ($F_5$). (c) Our proposed method: Targeted alignment of only the AIFI output feature ($F_5$), which possesses the highest-level semantics. This design allows gradients to backpropagate, enhancing both the AIFI module and the backbone.
• Figure 4: Comparison of dense features. We compare the feature map quality of DEIM-L (top) and RT-DETRv4-L (bottom) by projecting dense outputs to RGB space using PCA. The visualization reveals that our DSI module substantially enhances the semantic representation of AIFI features, which in turn benefits subsequent CCFF features. From left to right: input image, AIFI feature map $F_5$, and multi-scale CCFF features $P_3$, $P_4$, $P_5$.
• Figure 5: Validation AP evolution on COCO during training. We compare our dynamic GAM with a baseline model and several static $\lambda$ values for the DSI loss. The GAM strategy consistently outperforms all static configurations, showcasing its ability to provide stable and effective supervision throughout the training process.