Self-supervised Multiplex Consensus Mamba for General Image Fusion

TL;DR

SMC-Mamba tackles the challenge of general image fusion by integrating information from diverse modalities through three innovations: Modality-Agnostic Feature Enhancement (MAFE) for local-global feature synergy, Multiplex Consensus Cross-modal Mamba (MCCM) for dynamic, gated cross-modal fusion across multiple experts, and Bi-level Self-supervised Contrastive Learning Loss (BSCL) to preserve high-frequency details without extra computation. The MCCM introduces a gated Mixture-of-Experts framework with a time-decayed objective that balances exploration of diverse fusion strategies with later convergence to a unified representation, while cross-modal scanning strengthens inter-modal interactions. BSCL leverages Haar wavelet lifting to enforce alignment of high-frequency content at both feature and pixel levels, via $\mathcal{L}_{\text{fcl}}$ and $\mathcal{L}_{\text{pcl}}$, and the overall training optimizes $\mathcal{L}_{\text{total}} = \lambda_1 \mathcal{L}_{\text{fcl}} + \lambda_2 \mathcal{L}_{\text{pcl}} + \lambda_3 \mathcal{L}_{\text{mccm}} + \lambda_4 \mathcal{L}_{\text{ssim}} + \lambda_5 \mathcal{L}_{\text{int}}$. Empirically, SMC-Mamba achieves state-of-the-art performance across IVIF, MFIF, MDIF, MEIF tasks and improves downstream segmentation and detection, demonstrating strong generalization and practical impact for multi-modal vision tasks.

Abstract

Image fusion integrates complementary information from different modalities to generate high-quality fused images, thereby enhancing downstream tasks such as object detection and semantic segmentation. Unlike task-specific techniques that primarily focus on consolidating inter-modal information, general image fusion needs to address a wide range of tasks while improving performance without increasing complexity. To achieve this, we propose SMC-Mamba, a Self-supervised Multiplex Consensus Mamba framework for general image fusion. Specifically, the Modality-Agnostic Feature Enhancement (MAFE) module preserves fine details through adaptive gating and enhances global representations via spatial-channel and frequency-rotational scanning. The Multiplex Consensus Cross-modal Mamba (MCCM) module enables dynamic collaboration among experts, reaching a consensus to efficiently integrate complementary information from multiple modalities. The cross-modal scanning within MCCM further strengthens feature interactions across modalities, facilitating seamless integration of critical information from both sources. Additionally, we introduce a Bi-level Self-supervised Contrastive Learning Loss (BSCL), which preserves high-frequency information without increasing computational overhead while simultaneously boosting performance in downstream tasks. Extensive experiments demonstrate that our approach outperforms state-of-the-art (SOTA) image fusion algorithms in tasks such as infrared-visible, medical, multi-focus, and multi-exposure fusion, as well as downstream visual tasks.

Figures (4)

• Figure 1: The overall framework of our proposed network, which consists of three main components: 1) Modality-Agnostic Feature Enhancement module (MAFE). 2) Multiplex Consensus Cross-modal Mamba module (MCCM). 3) Bi-level Self-supervised Contrastive Learning Loss (BSCL).
• Figure 2: Visual comparisons of all the compared approaches on the MSRS dataset in IVIF task.
• Figure 3: Visual comparisons of all the compared approaches on the MFI-WHU dataset in MFIF task.
• Figure 4: Qualitative segmentation on the MSRS dataset.