EFSI-DETR: Efficient Frequency-Semantic Integration for Real-Time Small Object Detection in UAV Imagery

TL;DR

Real-time small object detection in UAV imagery is hindered by limited feature representation and ineffective multi-scale fusion. The paper introduces EFSI-DETR, which combines Dynamic Frequency-Spatial Unified Synergy Network (DyFusNet) for adaptive frequency-guided fusion and Efficient Semantic Feature Concentrator (ESFC) for lightweight, deep semantic extraction, augmented by a Fine-grained Feature Retention (FFR) strategy to preserve spatial detail. Key innovations include Dynamic Multi-resolution Spectral Decomposition (DMSD), Spatial–Frequency Cooperative Modulation (SFCM), Dynamic Expert Convolution, Efficient Ghost Blocks, and Dual-domain Guidance Aggregation, all designed to maintain real-time throughput on UAV data. Extensive experiments on VisDrone and CODrone show state-of-the-art AP and APs with high FPS (e.g., 188 FPS on RTX 4090) and strong ablation results validate the effectiveness of each component and design choice.

Abstract

Real-time small object detection in Unmanned Aerial Vehicle (UAV) imagery remains challenging due to limited feature representation and ineffective multi-scale fusion. Existing methods underutilize frequency information and rely on static convolutional operations, which constrain the capacity to obtain rich feature representations and hinder the effective exploitation of deep semantic features. To address these issues, we propose EFSI-DETR, a novel detection framework that integrates efficient semantic feature enhancement with dynamic frequency-spatial guidance. EFSI-DETR comprises two main components: (1) a Dynamic Frequency-Spatial Unified Synergy Network (DyFusNet) that jointly exploits frequency and spatial cues for robust multi-scale feature fusion, (2) an Efficient Semantic Feature Concentrator (ESFC) that enables deep semantic extraction with minimal computational cost. Furthermore, a Fine-grained Feature Retention (FFR) strategy is adopted to incorporate spatially rich shallow features during fusion to preserve fine-grained details, crucial for small object detection in UAV imagery. Extensive experiments on VisDrone and CODrone benchmarks demonstrate that our EFSI-DETR achieves the state-of-the-art performance with real-time efficiency, yielding improvement of \textbf{1.6}\% and \textbf{5.8}\% in AP and AP$_{s}$ on VisDrone, while obtaining \textbf{188} FPS inference speed on a single RTX 4090 GPU.

Figures (4)

• Figure 2: Overview of the proposed EFSI-DETR. DyFusNet exploits frequency and spatial cues to enable robust and adaptive multi-scale feature fusion. ESFC facilitates deep semantic representation with minimal computational overhead. FFR incorporates spatially rich shallow features during fusion, effectively preserving fine-grained details--crucial for small objects detection. Notably, the structures of the AIFI and Fusion block are consistent with what is employed in RT-DETRrtdetr.
• Figure 3: Visualization of detection results of RT-DETR and EFSI-DETR.
• Figure 4: Visualization of feature maps for variants composed of different components. The results demonstrate the progressive improvement in feature representation and object awareness with the integration of proposed components.
• Figure : (a)