Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Modality-Decoupled RGB-Thermal Object Detector via Query Fusion

Chao Tian, Zikun Zhou, Chao Yang, Guoqing Zhu, Fu'an Zhong, Zhenyu He

TL;DR

This work tackles RGB-T object detection under modality imbalance by introducing MDQF, a modality-decoupled detector that uses query fusion to transfer high-quality prompts between independent RGB and TIR DETR-like branches. It preserves the integrity of each single-modality backbone while enabling cross-modality refinement through top-$k$ query selection and adapter-based query adaptation, with a two-stage Separate-to-Joint training regimen that leverages unpaired data. Empirical results on FLIR and M3FD show state-of-the-art-like performance and robustness to modality degradation, validating both the method and its training strategy. The approach offers practical benefits in real-world scenarios where one modality may fail or be noisy, by maintaining detection capabilities through modality-independent branches and selective fusion.

Abstract

The advantage of RGB-Thermal (RGB-T) detection lies in its ability to perform modality fusion and integrate cross-modality complementary information, enabling robust detection under diverse illumination and weather conditions. However, under extreme conditions where one modality exhibits poor quality and disturbs detection, modality separation is necessary to mitigate the impact of noise. To address this problem, we propose a Modality-Decoupled RGB-T detection framework with Query Fusion (MDQF) to balance modality complementation and separation. In this framework, DETR-like detectors are employed as separate branches for the RGB and TIR images, with query fusion interspersed between the two branches in each refinement stage. Herein, query fusion is performed by feeding the high-quality queries from one branch to the other one after query selection and adaptation. This design effectively excludes the degraded modality and corrects the predictions using high-quality queries. Moreover, the decoupled framework allows us to optimize each individual branch with unpaired RGB or TIR images, eliminating the need for paired RGB-T data. Extensive experiments demonstrate that our approach delivers superior performance to existing RGB-T detectors and achieves better modality independence.

Modality-Decoupled RGB-Thermal Object Detector via Query Fusion

TL;DR

This work tackles RGB-T object detection under modality imbalance by introducing MDQF, a modality-decoupled detector that uses query fusion to transfer high-quality prompts between independent RGB and TIR DETR-like branches. It preserves the integrity of each single-modality backbone while enabling cross-modality refinement through top- query selection and adapter-based query adaptation, with a two-stage Separate-to-Joint training regimen that leverages unpaired data. Empirical results on FLIR and M3FD show state-of-the-art-like performance and robustness to modality degradation, validating both the method and its training strategy. The approach offers practical benefits in real-world scenarios where one modality may fail or be noisy, by maintaining detection capabilities through modality-independent branches and selective fusion.

Abstract

The advantage of RGB-Thermal (RGB-T) detection lies in its ability to perform modality fusion and integrate cross-modality complementary information, enabling robust detection under diverse illumination and weather conditions. However, under extreme conditions where one modality exhibits poor quality and disturbs detection, modality separation is necessary to mitigate the impact of noise. To address this problem, we propose a Modality-Decoupled RGB-T detection framework with Query Fusion (MDQF) to balance modality complementation and separation. In this framework, DETR-like detectors are employed as separate branches for the RGB and TIR images, with query fusion interspersed between the two branches in each refinement stage. Herein, query fusion is performed by feeding the high-quality queries from one branch to the other one after query selection and adaptation. This design effectively excludes the degraded modality and corrects the predictions using high-quality queries. Moreover, the decoupled framework allows us to optimize each individual branch with unpaired RGB or TIR images, eliminating the need for paired RGB-T data. Extensive experiments demonstrate that our approach delivers superior performance to existing RGB-T detectors and achieves better modality independence.
Paper Structure (11 sections, 6 equations, 3 figures, 4 tables)

This paper contains 11 sections, 6 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Overview of the Detection Framework
  4. Query Fusion
  5. Separate-to-Joint Model Learning
  6. Experiments
  7. Setup
  8. Comparison with State-of-the-art Methods
  9. Ablation Study
  10. Analysis of Modality-decoupling
  11. Conclusion

Figures (3)

  • Figure 1: Our proposed RGB-T detector equipped with the query fusion strategy for cross-modality information exchange. The framework keeps the independence of each branch, avoiding failure during the degradation of a modality. And each single-modality branch can be optimized separately.
  • Figure 2: (a) MDQF framework. Two standard DETR-like detectors are deployed to process RGB and TIR images independently. The continuous optimization of proposals in decoders of each modality is influenced by the other modality through the query fusion. (b) Query Fusion module. This is a practical implementation of the query fusion, including projection, selection, and collection of query vectors.
  • Figure 3: Qualitative results. Objects detected by the RGB branch are marked with bold red boxes in RGB images, and those from the TIR branch are marked with bold green boxes in TIR images.