M2I2HA: A Multi-modal Object Detection Method Based on Intra- and Inter-Modal Hypergraph Attention

TL;DR

M2I2HA tackles robust multi-modal object detection under challenging conditions by leveraging hypergraph attention to model high-order relationships within and across RGB and auxiliary modalities. The framework introduces two core modules, Intra-Hypergraph Enhancement and Inter-Hypergraph Fusion, along with an adaptive M2-FullPAD block to enable multi-level feature fusion while preserving important low-level details. Empirical results across DroneVehicle, FLIR-Aligned, LLVIP, and VEDAI demonstrate state-of-the-art performance and real-time inference, with ablations validating the contribution of each component. The work advances practical multi-modal perception by integrating structured high-order reasoning with efficient cross-modal alignment, and it points to future extensions with additional modalities such as depth or point clouds.

Abstract

Recent advances in multi-modal detection have significantly improved detection accuracy in challenging environments (e.g., low light, overexposure). By integrating RGB with modalities such as thermal and depth, multi-modal fusion increases data redundancy and system robustness. However, significant challenges remain in effectively extracting task-relevant information both within and across modalities, as well as in achieving precise cross-modal alignment. While CNNs excel at feature extraction, they are limited by constrained receptive fields, strong inductive biases, and difficulty in capturing long-range dependencies. Transformer-based models offer global context but suffer from quadratic computational complexity and are confined to pairwise correlation modeling. Mamba and other State Space Models (SSMs), on the other hand, are hindered by their sequential scanning mechanism, which flattens 2D spatial structures into 1D sequences, disrupting topological relationships and limiting the modeling of complex higher-order dependencies. To address these issues, we propose a multi-modal perception network based on hypergraph theory called M2I2HA. Our architecture includes an Intra-Hypergraph Enhancement module to capture global many-to-many high-order relationships within each modality, and an Inter-Hypergraph Fusion module to align, enhance, and fuse cross-modal features by bridging configuration and spatial gaps between data sources. We further introduce a M2-FullPAD module to enable adaptive multi-level fusion of multi-modal enhanced features within the network, meanwhile enhancing data distribution and flow across the architecture. Extensive object detection experiments on multiple public datasets against baselines demonstrate that M2I2HA achieves state-of-the-art performance in multi-modal object detection tasks.

Figures (13)

• Figure 1: Glare Resistance Comparison. Glare blurs the RGB, but does not affect the thermal.
• Figure 2: Feature discrepancies between visual and thermal modalities in different datasets. (a) Temporal capture misalignment. (b) Viewpoint-induced offset. (c) Configuration-based device inconsistency.
• Figure 3: Overall architecture of M2I2HA. M2I2HA adopts a YOLO-based architecture comprising a backbone, neck, and head. The backbone consists of two parallel feature extraction networks, two Intra-Hypergraph Enhancement modules, and an Inter-Hypergraph Fusion module, which respectively perform multi-modal feature extraction, intra-modal enhancement, and cross-modal fusion. The neck employs an FPN/PAN structure for multi-scale feature aggregation, while the head handles object detection. Additionally, the M2-FullPAD block optimizes multi-level feature information flow and enables dynamic fusion within the network.
• Figure 4: Architecture of the Intra-Hypergraph Enhancement Module and Its Sub-Blocks.
• Figure 5: Intre-Hypergraph fusion module