CoCoNet: Coupled Contrastive Learning Network with Multi-level Feature Ensemble for Multi-modality Image Fusion

TL;DR

CoCoNet addresses IVIF by explicitly learning cross-modal relationships through coupled contrastive losses that emphasize salient infrared targets and vivid visible textures. It integrates a self-adaptive weighting scheme that tunes SSIM and MSE contributions based on source image content, reducing the need for manually set fusion weights. A multi-level attention module and a VGG-based feature space support rich hierarchical representations, improving transmission and reducing information degeneration. The approach extends naturally to medical image fusion (MRI-PET, MRI-SPECT) and demonstrates strong task-driven performance in object detection and semantic segmentation, indicating practical impact for both surveillance and clinical imaging domains.

Abstract

Infrared and visible image fusion targets to provide an informative image by combining complementary information from different sensors. Existing learning-based fusion approaches attempt to construct various loss functions to preserve complementary features, while neglecting to discover the inter-relationship between the two modalities, leading to redundant or even invalid information on the fusion results. Moreover, most methods focus on strengthening the network with an increase in depth while neglecting the importance of feature transmission, causing vital information degeneration. To alleviate these issues, we propose a coupled contrastive learning network, dubbed CoCoNet, to realize infrared and visible image fusion in an end-to-end manner. Concretely, to simultaneously retain typical features from both modalities and to avoid artifacts emerging on the fused result, we develop a coupled contrastive constraint in our loss function. In a fused image, its foreground target / background detail part is pulled close to the infrared / visible source and pushed far away from the visible / infrared source in the representation space. We further exploit image characteristics to provide data-sensitive weights, allowing our loss function to build a more reliable relationship with source images. A multi-level attention module is established to learn rich hierarchical feature representation and to comprehensively transfer features in the fusion process. We also apply the proposed CoCoNet on medical image fusion of different types, e.g., magnetic resonance image, positron emission tomography image, and single photon emission computed tomography image. Extensive experiments demonstrate that our method achieves state-of-the-art (SOTA) performance under both subjective and objective evaluation, especially in preserving prominent targets and recovering vital textural details.

Figures (24)

• Figure 1: Visual illustration that highlights the existing issues in infrared and visible image fusion. Observe that the regions marked by yellow arrows signify redundant information, while those indicated by blue arrows denote information degradation. For example, DIDFuse is prone to retaining redundant information, as evidenced by the front windshield of the Jeep and the surrounding halo of light. On the other hand, MFEIF can result in the loss of essential information during its fusion process, as exemplified by the reduced visibility of clouds in the sky and the obscured indicator on the pole.
• Figure 2: Typical examples of salient mask $\mathcal{M}$ in TNO dataset.
• Figure 3: Overall architecture of CoCoNet.
• Figure 4: Architecture of channel attention.
• Figure 5: Schematic illustration of MIF tasks, where (a) pet/spect image, (b) Luminance (Y) channel of pet/spect image, (c) mri image, (d) Luminance channel (Y) of fused image, (e) fused image.