SMFusion: Semantic-Preserving Fusion of Multimodal Medical Images for Enhanced Clinical Diagnosis

TL;DR

SMFusion introduces a semantic-guided fusion framework for multimodal medical images that injects expert-level textual priors to preserve clinically relevant information and generate diagnostic reports. It combines a Restormer-based visual encoder, a semantic interaction alignment module, and a text injection mechanism to fuse images while maintaining semantic fidelity, guided by a medical semantic loss and gradient/reconstruction losses. A public multimodal medical image-text dataset and BiomedGPT-driven diagnostics demonstrate superior fusion quality and clinically meaningful reporting compared with seven state-of-the-art methods. The work advances clinically actionable image fusion by integrating medical knowledge, prompting-based guidance, and downstream diagnostic reporting, potentially improving diagnostic accuracy and workflow efficiency.

Abstract

Multimodal medical image fusion plays a crucial role in medical diagnosis by integrating complementary information from different modalities to enhance image readability and clinical applicability. However, existing methods mainly follow computer vision standards for feature extraction and fusion strategy formulation, overlooking the rich semantic information inherent in medical images. To address this limitation, we propose a novel semantic-guided medical image fusion approach that, for the first time, incorporates medical prior knowledge into the fusion process. Specifically, we construct a publicly available multimodal medical image-text dataset, upon which text descriptions generated by BiomedGPT are encoded and semantically aligned with image features in a high-dimensional space via a semantic interaction alignment module. During this process, a cross attention based linear transformation automatically maps the relationship between textual and visual features to facilitate comprehensive learning. The aligned features are then embedded into a text-injection module for further feature-level fusion. Unlike traditional methods, we further generate diagnostic reports from the fused images to assess the preservation of medical information. Additionally, we design a medical semantic loss function to enhance the retention of textual cues from the source images. Experimental results on test datasets demonstrate that the proposed method achieves superior performance in both qualitative and quantitative evaluations while preserving more critical medical information.

Figures (9)

• Figure 1: Comparison of Existing Medical Image Fusion Methods and Our Proposed Approach. (a) Existing fusion methods: These methods are primarily based on deep learning networks and integrate information from multiple source images through unsupervised learning strategies. (b) Proposed Semantic-Guided Image Fusion: Our approach integrates medical prior knowledge to explicitly guide the fusion process, establishing an interactive framework that ensures high-quality fusion results.
• Figure 2: Schematic illustration of multimodal medical image-text dataset construction. For each imaging modality, tailored text prompts are provided to guide BiomedGPT in generating corresponding descriptions.
• Figure 3: The workflow of SMFusion. It consists of three key components: the feature extraction module (a), the semantic interaction alignment module (b), and the text injection module (c). The encoded text features are guided through the L-layer (b) and (c) module to facilitate image fusion.
• Figure 4: Qualitative comparison of our SMFusion with seven state-of-the-art methods on three MRI and CT image pairs. Enlarged detail patches are highlighted with red and yellow boxes for visualization.
• Figure 5: Qualitative comparison of SMFusion with seven state-of-the-art methods on three MRI and PET image pairs. Notably, our method demonstrates superior performance in blood flow representation and texture preservation (e.g., the first row).