GAN-HA: A generative adversarial network with a novel heterogeneous dual-discriminator network and a new attention-based fusion strategy for infrared and visible image fusion

TL;DR

This paper tackles infrared-visible image fusion (IVIF) by introducing GAN-HA, a generative adversarial network with a heterogeneous dual-discriminator setup consisting of a salient discriminator for infrared intensity and a detailed discriminator for visible texture gradients. It employs an attention-based fusion strategy within the generator to adaptively weight information from the two sources, and uses a multi-scale skip-connected encoder–decoder to extract, fuse, and reconstruct the fused image. Across RoadScene, LLVIP, and M^3FD, GAN-HA achieves superior performance on standard fusion metrics, demonstrating robustness to noise and clear benefits for downstream tasks such as object detection and visual tracking, plus successful generalization to biomedical image fusion. The work highlights the effectiveness of modality-specific discriminators and attention-guided fusion for high-quality IVIF with strong practical potential.

Abstract

Infrared and visible image fusion (IVIF) aims to preserve thermal radiation information from infrared images while integrating texture details from visible images. Thermal radiation information is mainly expressed through image intensities, while texture details are typically expressed through image gradients. However, existing dual-discriminator generative adversarial networks (GANs) often rely on two structurally identical discriminators for learning, which do not fully account for the distinct learning needs of infrared and visible image information. To this end, this paper proposes a novel GAN with a heterogeneous dual-discriminator network and an attention-based fusion strategy (GAN-HA). Specifically, recognizing the intrinsic differences between infrared and visible images, we propose, for the first time, a novel heterogeneous dual-discriminator network to simultaneously capture thermal radiation information and texture details. The two discriminators in this network are structurally different, including a salient discriminator for infrared images and a detailed discriminator for visible images. They are able to learn rich image intensity information and image gradient information, respectively. In addition, a new attention-based fusion strategy is designed in the generator to appropriately emphasize the learned information from different source images, thereby improving the information representation ability of the fusion result. In this way, the fused images generated by GAN-HA can more effectively maintain both the salience of thermal targets and the sharpness of textures. Extensive experiments on various public datasets demonstrate the superiority of GAN-HA over other state-of-the-art (SOTA) algorithms while showcasing its higher potential for practical applications.

Figures (20)

• Figure 1: Example of the effect of IVIF.
• Figure 2: The framework of the proposed GAN-HA.
• Figure 3: Detailed structure of the generator.
• Figure 4: Construction of AFS.
• Figure 5: Details of discriminator networks.