Representation Space Constrained Learning with Modality Decoupling for Multimodal Object Detection
YiKang Shao, Tao Shi
TL;DR
This paper analyzes fusion degradation in multimodal object detection, identifying two core optimization defects: under-optimization of unimodal backbones due to fusion-induced gradient suppression, and imbalanced learning where weaker modalities are suppressed more than stronger ones. It introduces Representation Space Constrained Learning with Modality Decoupling (RSC-MD), combining a Representation Space Constraint (RSC) module with a Modality Decoupling (MD) module to amplify unimodal gradients and decouple modality optimization. The approach uses auxiliary detection heads and a gradient-mapping mechanism to enforce modality-specific learning while keeping fusion benefits, formalized with losses and update rules. Extensive experiments on FLIR, LLVIP, M3FD, and MFAD show consistent state-of-the-art gains with a lightweight parameter footprint, demonstrating strong practical impact for robust multimodal detection across challenging conditions.
Abstract
Multimodal object detection has attracted significant attention in both academia and industry for its enhanced robustness. Although numerous studies have focused on improving modality fusion strategies, most neglect fusion degradation, and none provide a theoretical analysis of its underlying causes. To fill this gap, this paper presents a systematic theoretical investigation of fusion degradation in multimodal detection and identifies two key optimization deficiencies: (1) the gradients of unimodal branch backbones are severely suppressed under multimodal architectures, resulting in under-optimization of the unimodal branches; (2) disparities in modality quality cause weaker modalities to experience stronger gradient suppression, which in turn results in imbalanced modality learning. To address these issues, this paper proposes a Representation Space Constrained Learning with Modality Decoupling (RSC-MD) method, which consists of two modules. The RSC module and the MD module are designed to respectively amplify the suppressed gradients and eliminate inter-modality coupling interference as well as modality imbalance, thereby enabling the comprehensive optimization of each modality-specific backbone. Extensive experiments conducted on the FLIR, LLVIP, M3FD, and MFAD datasets demonstrate that the proposed method effectively alleviates fusion degradation and achieves state-of-the-art performance across multiple benchmarks. The code and training procedures will be released at https://github.com/yikangshao/RSC-MD.