SGDFuse: SAM-Guided Diffusion for High-Fidelity Infrared and Visible Image Fusion

TL;DR

SGDFuse redefines infrared-visible image fusion as a semantic-guided generation task by fusing SAM-derived priors with a conditional diffusion model. The method employs a two-stage design: Stage I builds a robust structural prior $F_1$ via MSFEM and Transformer-based features, and Stage II refines this with a five-channel diffusion process that conditions on SAM masks $M_{ir}$ and $M_{vis}$. A Hierarchical Feature Aggregation Head (HFAH) and mask-guided losses ensure semantic consistency and edge fidelity, achieving state-of-the-art results across IVIF benchmarks and demonstrating strong generalization to medical fusion and high-level tasks like object detection and semantic segmentation. While introducing higher computational cost, SGDFuse offers a principled balance between semantic fidelity and visual quality, with robust performance under semantic perturbations and potential for real-time optimizations in the future.

Abstract

Infrared and visible image fusion (IVIF) aims to combine the thermal radiation information from infrared images with the rich texture details from visible images to enhance perceptual capabilities for downstream visual tasks. However, existing methods often fail to preserve key targets due to a lack of deep semantic understanding of the scene, while the fusion process itself can also introduce artifacts and detail loss, severely compromising both image quality and task performance. To address these issues, this paper proposes SGDFuse, a conditional diffusion model guided by the Segment Anything Model (SAM), to achieve high-fidelity and semantically-aware image fusion. The core of our method is to utilize high-quality semantic masks generated by SAM as explicit priors to guide the optimization of the fusion process via a conditional diffusion model. Specifically, the framework operates in a two-stage process: it first performs a preliminary fusion of multi-modal features, and then utilizes the semantic masks from SAM jointly with the preliminary fused image as a condition to drive the diffusion model's coarse-to-fine denoising generation. This ensures the fusion process not only has explicit semantic directionality but also guarantees the high fidelity of the final result. Extensive experiments demonstrate that SGDFuse achieves state-of-the-art performance in both subjective and objective evaluations, as well as in its adaptability to downstream tasks, providing a powerful solution to the core challenges in image fusion. The code of SGDFuse is available at https://github.com/boshizhang123/SGDFuse.

Figures (19)

• Figure 1: Comparison of different fusion methods on object detection and semantic segmentation tasks. Traditional methods perform well in terms of visual quality but are limited in downstream tasks, while the proposed method achieves better task performance while maintaining image quality.
• Figure 2: Visible and infrared images along with their corresponding SAM semantic masks. These semantic cues are used to guide the diffusion model optimization in the second stage.
• Figure 3: Visual Ablation for the Necessity of the Semantic-Guided Generation Paradigm. The red boxes show zoomed-in detail regions.
• Figure 4: Overall architecture of the SGDFuse framework. The framework consists of two stages: the first stage performs multi-modal feature extraction and generates an initial fused image; the second stage incorporates semantic prompts and a conditional diffusion network to refine structural details and enhance semantic consistency of the fused image.
• Figure 5: Architecture of the denoising network and the Hierarchical Feature Aggregation Head (HFAH).