YOLO-Master: MOE-Accelerated with Specialized Transformers for Enhanced Real-time Detection

TL;DR

This work tackles the inefficiency of static dense computation in real-time object detection by introducing YOLO-Master, which embeds an Efficient Sparse MoE (ES-MoE) into a YOLO-like pipeline. A lightweight Dynamic Routing Network selects a small subset of diverse multi-scale experts via Soft Top-K during training and Hard Top-K during inference, guided by a gating network and a load-balancing loss to prevent expert collapse. The approach yields state-of-the-art results across five benchmarks (notably 42.4% AP at 1.62 ms on COCO) while maintaining real-time speed, with pronounced gains on dense scenes and crowded scenes like SKU-110K and VisDrone. The method generalizes to downstream tasks such as image classification and segmentation, illustrating the versatility of conditional computation for efficient dense prediction in vision systems.

Abstract

Existing Real-Time Object Detection (RTOD) methods commonly adopt YOLO-like architectures for their favorable trade-off between accuracy and speed. However, these models rely on static dense computation that applies uniform processing to all inputs, misallocating representational capacity and computational resources such as over-allocating on trivial scenes while under-serving complex ones. This mismatch results in both computational redundancy and suboptimal detection performance. To overcome this limitation, we propose YOLO-Master, a novel YOLO-like framework that introduces instance-conditional adaptive computation for RTOD. This is achieved through a Efficient Sparse Mixture-of-Experts (ES-MoE) block that dynamically allocates computational resources to each input according to its scene complexity. At its core, a lightweight dynamic routing network guides expert specialization during training through a diversity enhancing objective, encouraging complementary expertise among experts. Additionally, the routing network adaptively learns to activate only the most relevant experts, thereby improving detection performance while minimizing computational overhead during inference. Comprehensive experiments on five large-scale benchmarks demonstrate the superiority of YOLO-Master. On MS COCO, our model achieves 42.4% AP with 1.62ms latency, outperforming YOLOv13-N by +0.8% mAP and 17.8% faster inference. Notably, the gains are most pronounced on challenging dense scenes, while the model preserves efficiency on typical inputs and maintains real-time inference speed. Code will be available.

Figures (4)

• Figure 1: Accuracy-latency trade-off on MS COCO. YOLO-Master-N achieves 42.4% AP at 1.62ms latency, outperforming baselines on the Pareto frontier.
• Figure 2: The framework of YOLO-Master. The architecture integrates ES-MoE modules into the Backbone and Neck for enhanced feature extraction and fusion. Input features are processed through a Dynamic Routing Network with Softmax Gating, selecting top-K experts for weighted aggregation. The framework adaptively switches between Standard, Soft Top-K (training), and Hard Top-K (inference) routing strategies for efficient multi-scale object detection across P3, P4, and P5 prediction layers.
• Figure 3: Loss ablation study across configurations. (a) DFL loss Comparison. (b) MoE Loss Comparison. (c) Validation mAP. (d) Total loss. (e) MoE loss Evolution. (f) mAP convergence.
• Figure 4: Qualitative comparison across four challenging scenarios. All test images are from the MS COCO lin2014microsoft ane PASCAL VOC. 2007+2012 everingham2010pascal test set