HSFusion: A high-level vision task-driven infrared and visible image fusion network via semantic and geometric domain transformation

TL;DR

The paper tackles the gap between semantic and geometric representations in infrared–visible image fusion to better support high‑level vision tasks. It introduces HSFusion, a two‑branch, CycleGAN‑based framework that simultaneously reinforces semantic segmentation and performs geometry‑aware fusion guided by semantic masks. Key contributions include dual non‑shared feature extractors, a semantic reinforce stage with cycle‑consistency constraints, an infrared segmentation‑driven fusion mask, and an adaptive fusion strategy validated on the FMB dataset. Experiments show state‑of‑the‑art performance in both fusion quality (SSIM, CC, PSNR, N^{AB/F}) and segmentation accuracy (IoU, mIoU), highlighting the practical impact for perception‑oriented and task‑driven multimodal fusion.

Abstract

Infrared and visible image fusion has been developed from vision perception oriented fusion methods to strategies which both consider the vision perception and high-level vision task. However, the existing task-driven methods fail to address the domain gap between semantic and geometric representation. To overcome these issues, we propose a high-level vision task-driven infrared and visible image fusion network via semantic and geometric domain transformation, terms as HSFusion. Specifically, to minimize the gap between semantic and geometric representation, we design two separate domain transformation branches by CycleGAN framework, and each includes two processes: the forward segmentation process and the reverse reconstruction process. CycleGAN is capable of learning domain transformation patterns, and the reconstruction process of CycleGAN is conducted under the constraint of these patterns. Thus, our method can significantly facilitate the integration of semantic and geometric information and further reduces the domain gap. In fusion stage, we integrate the infrared and visible features that extracted from the reconstruction process of two seperate CycleGANs to obtain the fused result. These features, containing varying proportions of semantic and geometric information, can significantly enhance the high level vision tasks. Additionally, we generate masks based on segmentation results to guide the fusion task. These masks can provide semantic priors, and we design adaptive weights for two distinct areas in the masks to facilitate image fusion. Finally, we conducted comparative experiments between our method and eleven other state-of-the-art methods, demonstrating that our approach surpasses others in both visual appeal and semantic segmentation task.

Figures (6)

• Figure 1: The problem formulation and network. The left part illustrates the interaction between image fusion and segmentation tasks in the latent space. The right part displays the simplified network architecture of HSFusion.
• Figure 2: The overall architecture of the proposed HSFusion. It comprises two main stages: the semantic reinforce stage and the fusion stage. The semantic reinforce stage encompasses two key processes: the segmentation process and the reconstruction process.
• Figure 3: The detailed architecture of the CGFE module. The central red section represents the overall architecture of the CGFE. The upper gray section is the details of generator $G$, which uses a U-net architecture. The lower yellow section outlines the architecture of generator $F$, and the left blue and right green sections respectively depict the detailed architectures of discriminators $D_x$ and $D_y$.
• Figure 4: Qualitative comparison of HSFusion with 11 state-of-the-art methods on the 00089 scene from the FMB dataset.
• Figure 5: Qualitative comparison of HSFusion with 11 state-of-the-art methods on the 00123 scene from the FMB dataset.