DIFF-MF: A Difference-Driven Channel-Spatial State Space Model for Multi-Modal Image Fusion

TL;DR

DIFF-MF tackles infrared–visible image fusion by introducing a difference-driven channel-spatial state-space framework that explicitly leverages modality discrepancies. It combines a difference-driven feature extractor with a cross-modal channel-exchange and a multiscale spatial-exchange to fuse features with linear computational complexity. The method achieves state-of-the-art performance on M$^{3}$FD, TNO, and DroneVehicle datasets and enhances downstream tasks such as object detection and semantic segmentation, while maintaining efficiency. This work demonstrates the strength of differential guidance and cross-modal state-space interactions for robust, real-time multimodal fusion.

Abstract

Multi-modal image fusion aims to integrate complementary information from multiple source images to produce high-quality fused images with enriched content. Although existing approaches based on state space model have achieved satisfied performance with high computational efficiency, they tend to either over-prioritize infrared intensity at the cost of visible details, or conversely, preserve visible structure while diminishing thermal target salience. To overcome these challenges, we propose DIFF-MF, a novel difference-driven channel-spatial state space model for multi-modal image fusion. Our approach leverages feature discrepancy maps between modalities to guide feature extraction, followed by a fusion process across both channel and spatial dimensions. In the channel dimension, a channel-exchange module enhances channel-wise interaction through cross-attention dual state space modeling, enabling adaptive feature reweighting. In the spatial dimension, a spatial-exchange module employs cross-modal state space scanning to achieve comprehensive spatial fusion. By efficiently capturing global dependencies while maintaining linear computational complexity, DIFF-MF effectively integrates complementary multi-modal features. Experimental results on the driving scenarios and low-altitude UAV datasets demonstrate that our method outperforms existing approaches in both visual quality and quantitative evaluation.

Figures (10)

• Figure 1: The feature visualization of different Mamba-based image fusion models. FusionMamba's over-fused features are dominated by infrared intensity and MambaDFuse's lack of interaction fails to highlight the thermal signatures of pedestrians. Our DIFF-MF effectively preserves thermal signatures from infrared modality and texture details from visible modality while maintaining optimal edge textures in background regions.
• Figure 2: The architecture of DIFF-MF. DIFF-MF consists of a difference driven feature extraction , a channel-exchange module, and a spatial-exchange module.
• Figure 3: The architecture of channel reweighting. Parameters $\alpha,\beta$ are learned from the ${F}_{vi}^{e}$ and ${F}_{ir}^{e}$. $\omega$ is generated by the Gate Generator.
• Figure 4: Qualitative comparisons of various methods on representative images selected from the M$^{3}$FD dataset.
• Figure 5: Qualitative comparisons of various methods on represen tative images selected from the TNO dataset.