Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Res2NetFuse: A Novel Res2Net-based Fusion Method for Infrared and Visible Images

Xu Song, Yongbiao Xiao, Hui Li, Xiao-Jun Wu, Jun Sun, Vasile Palade

TL;DR

This work introduces Res2NetFuse, a fusion framework for infrared and visible images that harnesses a Res2Net-based encoder to capture multi-scale features and an attention-enhanced fusion layer to improve feature integration. A lightweight, single-image training strategy trains the reconstruction model efficiently, achieving performance comparable to models trained on large datasets while significantly reducing training time. The fusion layer combines features via spatial attention mechanisms, yielding robust and artifact-free fused images. Extensive experiments on 20 IR-visible pairs show state-of-the-art performance across objective metrics and strong qualitative results, highlighting the method's practical impact for surveillance, remote sensing, and related imaging tasks.

Abstract

The fusion of visible light and infrared images has garnered significant attention in the field of imaging due to its pivotal role in various applications, including surveillance, remote sensing, and medical imaging. Therefore, this paper introduces a novel fusion framework using Res2Net architecture, capturing features across diverse receptive fields and scales for effective extraction of global and local features. Our methodology is structured into three fundamental components: the first part involves the Res2Net-based encoder, followed by the second part, which encompasses the fusion layer, and finally, the third part, which comprises the decoder. The encoder based on Res2Net is utilized for extracting multi-scale features from the input image. Simultaneously, with a single image as input, we introduce a pioneering training strategy tailored for a Res2Net-based encoder. We further enhance the fusion process with a novel strategy based on the attention model, ensuring precise reconstruction by the decoder for the fused image. Experimental results unequivocally showcase our method's unparalleled fusion performance, surpassing existing techniques, as evidenced by rigorous subjective and objective evaluations.

Res2NetFuse: A Novel Res2Net-based Fusion Method for Infrared and Visible Images

TL;DR

This work introduces Res2NetFuse, a fusion framework for infrared and visible images that harnesses a Res2Net-based encoder to capture multi-scale features and an attention-enhanced fusion layer to improve feature integration. A lightweight, single-image training strategy trains the reconstruction model efficiently, achieving performance comparable to models trained on large datasets while significantly reducing training time. The fusion layer combines features via spatial attention mechanisms, yielding robust and artifact-free fused images. Extensive experiments on 20 IR-visible pairs show state-of-the-art performance across objective metrics and strong qualitative results, highlighting the method's practical impact for surveillance, remote sensing, and related imaging tasks.

Abstract

The fusion of visible light and infrared images has garnered significant attention in the field of imaging due to its pivotal role in various applications, including surveillance, remote sensing, and medical imaging. Therefore, this paper introduces a novel fusion framework using Res2Net architecture, capturing features across diverse receptive fields and scales for effective extraction of global and local features. Our methodology is structured into three fundamental components: the first part involves the Res2Net-based encoder, followed by the second part, which encompasses the fusion layer, and finally, the third part, which comprises the decoder. The encoder based on Res2Net is utilized for extracting multi-scale features from the input image. Simultaneously, with a single image as input, we introduce a pioneering training strategy tailored for a Res2Net-based encoder. We further enhance the fusion process with a novel strategy based on the attention model, ensuring precise reconstruction by the decoder for the fused image. Experimental results unequivocally showcase our method's unparalleled fusion performance, surpassing existing techniques, as evidenced by rigorous subjective and objective evaluations.
Paper Structure (15 sections, 10 equations, 5 figures, 4 tables)

This paper contains 15 sections, 10 equations, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. DenseFuse
  4. Res2Net
  5. Methodology
  6. The Proposed Fusion Method
  7. Training
  8. The Fusion Layer
  9. Experiments
  10. Environment and Experimental Settings
  11. Training Strategy
  12. Reconstruction Comparisons
  13. Fusion Comparisons
  14. Assessment from Subjective and Objective Perspectives
  15. Conclusion

Figures (5)

  • Figure 1: DenseFuse Architecture.
  • Figure 2: Res2Net module.
  • Figure 3: The presented framework is tailored for grayscale images. $I_1$ and $I_2$ mean the source image, both of which undergo encoding to extract features at different scales. Subsequently, in the fusion layer, these extracted features are fused before being fed into the decoder. Ultimately, the output of the decoder is the fused image denoted as $f$.
  • Figure 4: Loss graphs. (a) $L_{pixel}$; (b) $L_{ssim}$; (c) Total loss.
  • Figure 5: Experimental findings: (a) The infrared image (b) The visible light image (c) JSRSD (d) DCHWT (e) JSR (f) FusionGAN (g) DenseFuse (h) AEFusion (i) AT-GAN (j) CUED (k) MUFusion (l) MSDNet utilizing the mean fusion strategy (m) MSDNet utilizing the $l_1$-norm fusion strategy (n) Res2NetFuse utilizing the mean fusion strategy (o) Res2NetFuse utilizing the $l_1$-norm fusion strategy.