Mixture of Scale Experts for Alignment-free RGBT Video Object Detection and A Unified Benchmark

TL;DR

This work tackles the challenge of alignment-free RGBT video object detection by introducing MSENet, a one-stage detector that fuses RGB and thermal features through a Mixture of Scale Experts and deformable convolution, enabling robust cross-modal alignment without manual image pairing. A Temporal Information Injection Module further integrates temporal context, and a dedicated loss framework combines classification and regression objectives. The authors also provide UVT-VOD2024, a large, diverse alignment-free benchmark with 60,988 images and 271,835 object instances across 11 scene categories to evaluate realism and robustness. Empirical results show MSENet achieves notable AP_{50} gains (e.g., +4.2% over prior SOTA on UVT-VOD2024) and high FPS (131) on UVT-VOD2024, while demonstrating strong generalization to aligned data (VT-VOD50) with configurable expert scales, highlighting practical impact for real-world multimodal VOD systems. Overall, the approach advances alignment-free multimodal VOD and provides a valuable dataset for reproducible research and benchmarking in multispectral video analysis.

Abstract

Existing RGB-Thermal Video Object Detection (RGBT VOD) methods predominantly rely on the manual alignment of image pairs, that is both labor-intensive and time-consuming. This dependency significantly restricts the scalability and practical applicability of these methods in real-world scenarios. To address this critical limitation, we propose a novel framework termed the Mixture of Scale Experts Network (MSENet). MSENet integrates multiple experts trained at different perceptual scales, enabling the capture of scale discrepancies between RGB and thermal image pairs without the need for explicit alignment. Specifically, to address the issue of unaligned scales, MSENet introduces a set of experts designed to perceive the correlation between RGBT image pairs across various scales. These experts are capable of identifying and quantifying the scale differences inherent in the image pairs. Subsequently, a dynamic routing mechanism is incorporated to assign adaptive weights to each expert, allowing the network to dynamically select the most appropriate experts based on the specific characteristics of the input data. Furthermore, to address the issue of weakly unaligned positions, we integrate deformable convolution into the network. Deformable convolution is employed to learn position displacements between the RGB and thermal modalities, thereby mitigating the impact of spatial misalignment. To provide a comprehensive evaluation platform for alignment-free RGBT VOD, we introduce a new benchmark dataset. This dataset includes eleven common object categories, with a total of 60,988 images and 271,835 object instances. The dataset encompasses a wide range of scenes from both daily life and natural environments, ensuring high content diversity and complexity.

Figures (9)

• Figure 1: Multispectral sensors capture images across different wavelengths, leading to significant differences in the scale and field of view of objects. These discrepancies stem from the distinct optical properties and sensor characteristics associated with each wavelength band, resulting in variations in spatial resolution and coverage area.
• Figure 2: Architecture diagram of MSENet. Our objective is to enhance the features of the current frame $Frame_{t}^{RGB}$; therefore, we do not utilize $Frame_{t-1}^{Thermal}$ for training. This decision is based on the inherent differences in semantics, spatial context, and temporal factors between $Frame_{t}^{RGB}$ and $Frame_{t-1}^{Thermal}$. Such disparities can negatively affect the fusion effect and reduce operational efficiency.
• Figure 3: Flowchart of MSE module. Where "DConv" represents the deformable convolution operation.
• Figure 4: Flowchart of Routing in MSE.
• Figure 5: Flowchart of TIIM.