Modality Dominance-Aware Optimization for Embodied RGB-Infrared Perception

TL;DR

The paper tackles optimization bias in RGB–IR object detection arising from asymmetric modality information by introducing Modality Dominance Index (MDI) to quantify dominance during training. It then proposes Modality Dominance-Aware Cross-modal Learning (MDACL), combining Hierarchical Cross-modal Guidance (HCG) for multi-level alignment with Adversarial Equilibrium Regularization (AER) to balance fusion dynamics. Key innovations include a two-stage cross-modal guidance mechanism and a semantic distillation framework with adaptive teacher signals and gradient-preserving constraints. Extensive experiments on LLVIP, M3FD, and FLIR demonstrate state-of-the-art performance and robust, balanced cross-modal fusion under challenging conditions, underscoring the method’s practical impact for embodied multimodal perception.

Abstract

RGB-Infrared (RGB-IR) multimodal perception is fundamental to embodied multimedia systems operating in complex physical environments. Although recent cross-modal fusion methods have advanced RGB-IR detection, the optimization dynamics caused by asymmetric modality characteristics remain underexplored. In practice, disparities in information density and feature quality introduce persistent optimization bias, leading training to overemphasize a dominant modality and hindering effective fusion. To quantify this phenomenon, we propose the Modality Dominance Index (MDI), which measures modality dominance by jointly modeling feature entropy and gradient contribution. Based on MDI, we develop a Modality Dominance-Aware Cross-modal Learning (MDACL) framework that regulates cross-modal optimization. MDACL incorporates Hierarchical Cross-modal Guidance (HCG) to enhance feature alignment and Adversarial Equilibrium Regularization (AER) to balance optimization dynamics during fusion. Extensive experiments on three RGB-IR benchmarks demonstrate that MDACL effectively mitigates optimization bias and achieves SOTA performance.

Figures (5)

• Figure 1: Illustration of the optimization bias phenomenon in RGB–Infrared detection. (a) Performance comparison on the M3FD dataset under different optimization settings, including the RGB+IR joint training baseline ($\lambda\!=\!1$) and two variants with manually amplified RGB gradients ($\lambda\!=\!3$ and $\lambda\!=\!5$). (b) Inference performance on M3FD using RGB-only, IR-only, and RGB+IR inputs for the baseline RGB+IR model and the RGB Grad-Boost ($\lambda\!=\!5$) variant. (c) Average gradient contribution and object-region feature entropy of RGB and IR modalities during joint training on M3FD.
• Figure 2: Overview of the proposed MDACL framework. RGB and IR images are processed by a dual-stream backbone, followed by (a) Modality Dominance Index (MDI) to estimate modality dominance. The dominance scores guide (b) Hierarchical Cross-modal Guidance (HCG) for cross-modal feature alignment, and (c) Adversarial Equilibrium Regularization (AER) for balanced feature fusion and stable optimization.
• Figure 3: Visualization of some RGB-Infrared detection methods on M3FD and FLIR. (a)-(c) present the results of M3FD dataset, and (d)-(f) present the results of FLIR dataset. The targets encircled by yellow ellipses are false positives, while those encircled by red ellipses are missed detections.
• Figure 4: Gradient bias comparison and its impact on model performance. (a) Performance versus average gradient bias for four RGB–IR detectors. (b) Evolution of average gradient bias across different training epochs.
• Figure 5: Comparison of three modality weighting strategies on LLVIP: Forward (forward weighting), Uniform (uniform weighting used as the baseline), and Inverse (inverse weighting proposed in our work).