MATCNN: Infrared and Visible Image Fusion Method Based on Multi-scale CNN with Attention Transformer

TL;DR

MATCNN tackles cross-modal infrared–visible image fusion by jointly modeling multi-scale local features and global features. The framework combines a multi-scale fusion module (MSFM) with a global feature extraction module (GFEM) based on an attention Transformer, connected through a fusion mechanism and guided by a salient-information mask. A novel loss blends content, SSIM, and a four-scale global feature term to preserve salient infrared details, background texture, and global feature continuity. Experimental results on TNO, MSRS, and RoadScene demonstrate improved saliency, texture preservation, and information retention over competitive methods, with code to reproduce results released publicly. The approach offers a practical path toward robust cross-modal fusion with strong generalization across datasets.

Abstract

While attention-based approaches have shown considerable progress in enhancing image fusion and addressing the challenges posed by long-range feature dependencies, their efficacy in capturing local features is compromised by the lack of diverse receptive field extraction techniques. To overcome the shortcomings of existing fusion methods in extracting multi-scale local features and preserving global features, this paper proposes a novel cross-modal image fusion approach based on a multi-scale convolutional neural network with attention Transformer (MATCNN). MATCNN utilizes the multi-scale fusion module (MSFM) to extract local features at different scales and employs the global feature extraction module (GFEM) to extract global features. Combining the two reduces the loss of detail features and improves the ability of global feature representation. Simultaneously, an information mask is used to label pertinent details within the images, aiming to enhance the proportion of preserving significant information in infrared images and background textures in visible images in fused images. Subsequently, a novel optimization algorithm is developed, leveraging the mask to guide feature extraction through the integration of content, structural similarity index measurement, and global feature loss. Quantitative and qualitative evaluations are conducted across various datasets, revealing that MATCNN effectively highlights infrared salient targets, preserves additional details in visible images, and achieves better fusion results for cross-modal images. The code of MATCNN will be available at https://github.com/zhang3849/MATCNN.git.

Figures (18)

• Figure 1: Overview framework of the proposed MATCNN method.
• Figure 2: The specific structure of the GFEM.
• Figure 3: The overall loss function of MATCNN.
• Figure 4: Fusion results with different $\alpha$ and $\gamma$.
• Figure 5: Fusion results of soldiers$\_$with$\_$jeep in the TNO dataset.