COMO: Cross-Mamba Interaction and Offset-Guided Fusion for Multimodal Object Detection

TL;DR

COMO tackles misalignment and efficiency challenges in multimodal object detection by introducing Cross-Mamba interaction and Offset-guided fusion. It leverages high-level features for inter-modal interaction, incorporates a Global and Local Scan within a cross-Mamba framework to capture global context and local correlations, and uses an Offset-Guided Fusion module to preserve low-level details while mitigating offsets. The approach yields state-of-the-art results on DroneVehicle, LLVIP, and VEDAI with lower FLOPs and faster inference, demonstrating strong practical relevance for aerial, drone, and road surveillance. Overall, COMO advances efficient, robust multimodal detection in remote sensing and related applications by synergistically combining sequence-based interaction, local/global context, and offset-aware fusion.

Abstract

Single-modal object detection tasks often experience performance degradation when encountering diverse scenarios. In contrast, multimodal object detection tasks can offer more comprehensive information about object features by integrating data from various modalities. Current multimodal object detection methods generally use various fusion techniques, including conventional neural networks and transformer-based models, to implement feature fusion strategies and achieve complementary information. However, since multimodal images are captured by different sensors, there are often misalignments between them, making direct matching challenging. This misalignment hinders the ability to establish strong correlations for the same object across different modalities. In this paper, we propose a novel approach called the CrOss-Mamba interaction and Offset-guided fusion (COMO) framework for multimodal object detection tasks. The COMO framework employs the cross-mamba technique to formulate feature interaction equations, enabling multimodal serialized state computation. This results in interactive fusion outputs while reducing computational overhead and improving efficiency. Additionally, COMO leverages high-level features, which are less affected by misalignment, to facilitate interaction and transfer complementary information between modalities, addressing the positional offset challenges caused by variations in camera angles and capture times. Furthermore, COMO incorporates a global and local scanning mechanism in the cross-mamba module to capture features with local correlation, particularly in remote sensing images. To preserve low-level features, the offset-guided fusion mechanism ensures effective multiscale feature utilization, allowing the construction of a multiscale fusion data cube that enhances detection performance.

Figures (12)

• Figure 1: The phenomenon of offset in multimodal images. (a) Specific scenarios in multimodal data acquisition. (b) Offset due to differences in capture time. (c) Offset due to differences in capture angles.
• Figure 2: Offset statistics results using the DroneVehicle dataset as an example. (a) Overview of data offsets. (b) Specific offset level statistics.
• Figure 3: Architecture of COMO framework. The framework consists of three main components: Mamba Interaction Block, Global and Local Scan method, and Offset-Guided Fusion. The Mamba Interaction Block is used to extract high-level features and perform inter-modal interaction. The Global and Local Scan method is used to strengthen the local feature association. The Offset-Guided Fusion module is used to fuse high-level features and low-level features.
• Figure 4: Mamba interaction block. The block consists of two modules: (a) single-mamba block and (b) cross-mamba block. The single-mamba block is used to extract features from single-modal data, while the cross-mamba block is used to interact between multimodal data.
• Figure 5: Different scanning mechanisms. (a) Global scanning. (b) Local scanning.