Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CoMoFusion: Fast and High-quality Fusion of Infrared and Visible Image with Consistency Model

Zhiming Meng, Hui Li, Zeyang Zhang, Zhongwei Shen, Yunlong Yu, Xiaoning Song, Xiaojun Wu

TL;DR

CoMoFusion tackles the instability of GAN-based IVF and the slow sampling of diffusion models by introducing a consistency-model backbone to fuse infrared and visible images. The method constructs multi-modal joint features with forward and reverse consistency processes, then feeds these features into a tailored fusion module guided by a novel pixel-value selection loss, $L_{pvs}$, and a gradient loss, $L_{grad}$. Ablation studies and broad experiments on KAIST, TNO, and MSRS show state-of-the-art fusion quality with fast inference, validating the approach's robustness and real-time potential. Overall, CoMoFusion provides a stable, fast, and high-fidelity IVF framework with practical impact for downstream vision tasks.

Abstract

Generative models are widely utilized to model the distribution of fused images in the field of infrared and visible image fusion. However, current generative models based fusion methods often suffer from unstable training and slow inference speed. To tackle this problem, a novel fusion method based on consistency model is proposed, termed as CoMoFusion, which can generate the high-quality images and achieve fast image inference speed. In specific, the consistency model is used to construct multi-modal joint features in the latent space with the forward and reverse process. Then, the infrared and visible features extracted by the trained consistency model are fed into fusion module to generate the final fused image. In order to enhance the texture and salient information of fused images, a novel loss based on pixel value selection is also designed. Extensive experiments on public datasets illustrate that our method obtains the SOTA fusion performance compared with the existing fusion methods.

CoMoFusion: Fast and High-quality Fusion of Infrared and Visible Image with Consistency Model

TL;DR

CoMoFusion tackles the instability of GAN-based IVF and the slow sampling of diffusion models by introducing a consistency-model backbone to fuse infrared and visible images. The method constructs multi-modal joint features with forward and reverse consistency processes, then feeds these features into a tailored fusion module guided by a novel pixel-value selection loss, , and a gradient loss, . Ablation studies and broad experiments on KAIST, TNO, and MSRS show state-of-the-art fusion quality with fast inference, validating the approach's robustness and real-time potential. Overall, CoMoFusion provides a stable, fast, and high-fidelity IVF framework with practical impact for downstream vision tasks.

Abstract

Generative models are widely utilized to model the distribution of fused images in the field of infrared and visible image fusion. However, current generative models based fusion methods often suffer from unstable training and slow inference speed. To tackle this problem, a novel fusion method based on consistency model is proposed, termed as CoMoFusion, which can generate the high-quality images and achieve fast image inference speed. In specific, the consistency model is used to construct multi-modal joint features in the latent space with the forward and reverse process. Then, the infrared and visible features extracted by the trained consistency model are fed into fusion module to generate the final fused image. In order to enhance the texture and salient information of fused images, a novel loss based on pixel value selection is also designed. Extensive experiments on public datasets illustrate that our method obtains the SOTA fusion performance compared with the existing fusion methods.
Paper Structure (26 sections, 10 equations, 6 figures, 4 tables)

This paper contains 26 sections, 10 equations, 6 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Deep Learning based IVF Methods
  4. Consistency Model
  5. Method
  6. Construct Multi-modal Joint Features
  7. Forward Process.
  8. Reverse Process.
  9. Loss Function of Consistency Model Training Phase.
  10. Image Fusion with Consistency Features
  11. Multi-modal Joint Features.
  12. Fusion Module.
  13. Loss Function of Fusion Module Training Phase.
  14. Experiments
  15. Setup
  16. ...and 11 more sections

Figures (6)

  • Figure 1: (a) The workflow of existing GAN-based methods (GANMcC ma2020ganmcc). (b) The workflow of existing DDPM-based methods (DDFM zhao2023ddfm). (c) The workflow of our method.
  • Figure 2: The framework of CoMoFusion. (a)Two training stages: consistency training, fusion training. (b)The forward and reverse process of consistency model training. (c)The loss function of fusion training.
  • Figure 3: The structure of consistency model and fusion module.
  • Figure 4: Qualitative comparison of the image “35” in the TNO dataset.
  • Figure 5: Qualitative comparison of the image “00838N” in the MSRS dataset.
  • ...and 1 more figures