Towards a Generalizable Fusion Architecture for Multimodal Object Detection

TL;DR

The paper tackles robust multimodal object detection with RGB and infrared data under challenging conditions. It introduces FMCAF, a generalizable preprocessing fusion framework that combines a learnable frequency-domain Freq-Filter with a Multimodal Cross Attention Fusion (MCAF) module to enable intermodal feature sharing before detection. Key contributions include validating a dataset-agnostic fusion approach across VEDAI and LLVIP, demonstrating large gains over simple concatenation (e.g., $+13.9\%$ in $mAP@50$ on VEDAI and $+1.1\%$ on LLVIP) and showing how a learnable mixing coefficient α governs the balance between raw and filtered inputs via the blending rule $X_{blend} = α X_{filtered} + (1-α) X$. The results indicate FMCAF as a practical foundation for robust multimodal fusion with potential for extension to segmentation and edge deployment.

Abstract

Multimodal object detection improves robustness in chal- lenging conditions by leveraging complementary cues from multiple sensor modalities. We introduce Filtered Multi- Modal Cross Attention Fusion (FMCAF), a preprocess- ing architecture designed to enhance the fusion of RGB and infrared (IR) inputs. FMCAF combines a frequency- domain filtering block (Freq-Filter) to suppress redun- dant spectral features with a cross-attention-based fusion module (MCAF) to improve intermodal feature sharing. Unlike approaches tailored to specific datasets, FMCAF aims for generalizability, improving performance across different multimodal challenges without requiring dataset- specific tuning. On LLVIP (low-light pedestrian detec- tion) and VEDAI (aerial vehicle detection), FMCAF outper- forms traditional fusion (concatenation), achieving +13.9% mAP@50 on VEDAI and +1.1% on LLVIP. These results support the potential of FMCAF as a flexible foundation for robust multimodal fusion in future detection pipelines.

Figures (3)

• Figure 1: Overview of the proposed architecture. The Freq-Filter module (left) applies frequency-domain filtering to each modality $m \in \{\text{RGB}, \text{IR}\}$, while the MCAF block (right) performs attention-based fusion using both self and cross-attention mechanisms.
• Figure 2: Qualitative comparison of detection results across fusion methods. Each row shows one input scene, and each column presents the detection output from a different fusion configuration. Orange circles highlight missed vehicle detections.
• Figure 3: Evolution of $\alpha$ during training