Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Text-DiFuse: An Interactive Multi-Modal Image Fusion Framework based on Text-modulated Diffusion Model

Hao Zhang, Lei Cao, Jiayi Ma

TL;DR

A novel interactive multi-modal image fusion framework based on the text-modulated diffusion model, called Text-DiFuse, which is the first attempt to deeply and explicitly embed information fusion within the diffusion process, effectively addressing compound degradation in image fusion.

Abstract

Existing multi-modal image fusion methods fail to address the compound degradations presented in source images, resulting in fusion images plagued by noise, color bias, improper exposure, \textit{etc}. Additionally, these methods often overlook the specificity of foreground objects, weakening the salience of the objects of interest within the fused images. To address these challenges, this study proposes a novel interactive multi-modal image fusion framework based on the text-modulated diffusion model, called Text-DiFuse. First, this framework integrates feature-level information integration into the diffusion process, allowing adaptive degradation removal and multi-modal information fusion. This is the first attempt to deeply and explicitly embed information fusion within the diffusion process, effectively addressing compound degradation in image fusion. Second, by embedding the combination of the text and zero-shot location model into the diffusion fusion process, a text-controlled fusion re-modulation strategy is developed. This enables user-customized text control to improve fusion performance and highlight foreground objects in the fused images. Extensive experiments on diverse public datasets show that our Text-DiFuse achieves state-of-the-art fusion performance across various scenarios with complex degradation. Moreover, the semantic segmentation experiment validates the significant enhancement in semantic performance achieved by our text-controlled fusion re-modulation strategy. The code is publicly available at https://github.com/Leiii-Cao/Text-DiFuse.

Text-DiFuse: An Interactive Multi-Modal Image Fusion Framework based on Text-modulated Diffusion Model

TL;DR

A novel interactive multi-modal image fusion framework based on the text-modulated diffusion model, called Text-DiFuse, which is the first attempt to deeply and explicitly embed information fusion within the diffusion process, effectively addressing compound degradation in image fusion.

Abstract

Existing multi-modal image fusion methods fail to address the compound degradations presented in source images, resulting in fusion images plagued by noise, color bias, improper exposure, \textit{etc}. Additionally, these methods often overlook the specificity of foreground objects, weakening the salience of the objects of interest within the fused images. To address these challenges, this study proposes a novel interactive multi-modal image fusion framework based on the text-modulated diffusion model, called Text-DiFuse. First, this framework integrates feature-level information integration into the diffusion process, allowing adaptive degradation removal and multi-modal information fusion. This is the first attempt to deeply and explicitly embed information fusion within the diffusion process, effectively addressing compound degradation in image fusion. Second, by embedding the combination of the text and zero-shot location model into the diffusion fusion process, a text-controlled fusion re-modulation strategy is developed. This enables user-customized text control to improve fusion performance and highlight foreground objects in the fused images. Extensive experiments on diverse public datasets show that our Text-DiFuse achieves state-of-the-art fusion performance across various scenarios with complex degradation. Moreover, the semantic segmentation experiment validates the significant enhancement in semantic performance achieved by our text-controlled fusion re-modulation strategy. The code is publicly available at https://github.com/Leiii-Cao/Text-DiFuse.

Paper Structure

This paper contains 21 sections, 14 equations, 8 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Deep Multi-modal Image Fusion.
  4. Diffusion Model.
  5. Zero-shot Location.
  6. Methodology
  7. Problem Statement and Modeling
  8. Explicit Coupling Paradigm of Information Fusion and Diffusion
  9. Diffusion for Degradation Removal.
  10. Fusion Control Module for Information Fusion.
  11. Diffusion Fusion.
  12. Text-controlled Fusion Re-modulation Strategy
  13. Experiments
  14. Configuration.
  15. Comparative Experiments.
  16. ...and 6 more sections

Figures (8)

  • Figure 1: Our proposed explicit coupling paradigm of multi-modal information fusion and diffusion.
  • Figure 2: The pipeline of our Text-DiFuse. (a) The text-controlled diffusion fusion process; (b) training diffusion model for degradation removal; (c) the detailed structure of fusion control module.
  • Figure 3: Visual comparison of image fusion methods.
  • Figure 4: Visual comparison of enhancement plus image fusion methods.
  • Figure 5: Visual results of generalization evaluation.
  • ...and 3 more figures