Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DFVO: Learning Darkness-free Visible and Infrared Image Disentanglement and Fusion All at Once

Qi Zhou, Yukai Shi, Xiaojun Yang, Xiaoyu Xian, Lunjia Liao, Ruimao Zhang, Liang Lin

TL;DR

This work tackles the degradation of visible–infrared fusion under low-light conditions by proposing DFVO, a darkness-free network that learns illumination enhancement and image fusion in a single cascaded multi-task framework. Central to DFVO are the latent-common feature extractor (LCFE), the details-extraction module (DEM), the hyper cross-attention module (HCAM), and a carefully designed multi-term loss that preserves high-frequency details, textures, and color fidelity while handling illumination disentanglement via a Retinex-based approach. The method demonstrates superior performance on the LLVIP dataset, achieving state-of-the-art PSNR and correlation metrics, clearer textures, and more balanced illumination, as well as improved detection results on downstream tasks. The results suggest DFVO's practical value for autonomous driving and other high-level vision applications where multi-modal fusion under challenging lighting is crucial, while acknowledging computational considerations and avenues for lighter-weight refinements.

Abstract

Visible and infrared image fusion is one of the most crucial tasks in the field of image fusion, aiming to generate fused images with clear structural information and high-quality texture features for high-level vision tasks. However, when faced with severe illumination degradation in visible images, the fusion results of existing image fusion methods often exhibit blurry and dim visual effects, posing major challenges for autonomous driving. To this end, a Darkness-Free network is proposed to handle Visible and infrared image disentanglement and fusion all at Once (DFVO), which employs a cascaded multi-task approach to replace the traditional two-stage cascaded training (enhancement and fusion), addressing the issue of information entropy loss caused by hierarchical data transmission. Specifically, we construct a latent-common feature extractor (LCFE) to obtain latent features for the cascaded tasks strategy. Firstly, a details-extraction module (DEM) is devised to acquire high-frequency semantic information. Secondly, we design a hyper cross-attention module (HCAM) to extract low-frequency information and preserve texture features from source images. Finally, a relevant loss function is designed to guide the holistic network learning, thereby achieving better image fusion. Extensive experiments demonstrate that our proposed approach outperforms state-of-the-art alternatives in terms of qualitative and quantitative evaluations. Particularly, DFVO can generate clearer, more informative, and more evenly illuminated fusion results in the dark environments, achieving best performance on the LLVIP dataset with 63.258 dB PSNR and 0.724 CC, providing more effective information for high-level vision tasks. Our code is publicly accessible at https://github.com/DaVin-Qi530/DFVO.

DFVO: Learning Darkness-free Visible and Infrared Image Disentanglement and Fusion All at Once

TL;DR

This work tackles the degradation of visible–infrared fusion under low-light conditions by proposing DFVO, a darkness-free network that learns illumination enhancement and image fusion in a single cascaded multi-task framework. Central to DFVO are the latent-common feature extractor (LCFE), the details-extraction module (DEM), the hyper cross-attention module (HCAM), and a carefully designed multi-term loss that preserves high-frequency details, textures, and color fidelity while handling illumination disentanglement via a Retinex-based approach. The method demonstrates superior performance on the LLVIP dataset, achieving state-of-the-art PSNR and correlation metrics, clearer textures, and more balanced illumination, as well as improved detection results on downstream tasks. The results suggest DFVO's practical value for autonomous driving and other high-level vision applications where multi-modal fusion under challenging lighting is crucial, while acknowledging computational considerations and avenues for lighter-weight refinements.

Abstract

Visible and infrared image fusion is one of the most crucial tasks in the field of image fusion, aiming to generate fused images with clear structural information and high-quality texture features for high-level vision tasks. However, when faced with severe illumination degradation in visible images, the fusion results of existing image fusion methods often exhibit blurry and dim visual effects, posing major challenges for autonomous driving. To this end, a Darkness-Free network is proposed to handle Visible and infrared image disentanglement and fusion all at Once (DFVO), which employs a cascaded multi-task approach to replace the traditional two-stage cascaded training (enhancement and fusion), addressing the issue of information entropy loss caused by hierarchical data transmission. Specifically, we construct a latent-common feature extractor (LCFE) to obtain latent features for the cascaded tasks strategy. Firstly, a details-extraction module (DEM) is devised to acquire high-frequency semantic information. Secondly, we design a hyper cross-attention module (HCAM) to extract low-frequency information and preserve texture features from source images. Finally, a relevant loss function is designed to guide the holistic network learning, thereby achieving better image fusion. Extensive experiments demonstrate that our proposed approach outperforms state-of-the-art alternatives in terms of qualitative and quantitative evaluations. Particularly, DFVO can generate clearer, more informative, and more evenly illuminated fusion results in the dark environments, achieving best performance on the LLVIP dataset with 63.258 dB PSNR and 0.724 CC, providing more effective information for high-level vision tasks. Our code is publicly accessible at https://github.com/DaVin-Qi530/DFVO.
Paper Structure (17 sections, 25 equations, 9 figures, 6 tables, 1 algorithm)

This paper contains 17 sections, 25 equations, 9 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. DL-based Image Fusion
  4. Invertible Neural Networks
  5. Self-Attention Mechanisms
  6. Proposed Method
  7. Learning Cascaded Tasks All at Once
  8. Latent-Common Feature Extractor
  9. Loss Function
  10. Experiments
  11. Experimental Configurations
  12. Performance Analysis on Image Fusion Tasks
  13. Generalization Analysis on Image Fusion Tasks
  14. Ablation Experiment
  15. Performance Analysis on Detection Tasks
  16. ...and 2 more sections

Figures (9)

  • Figure 1: Existing fusion methods in low-light environments vs. DFVO. (a) The workflow of single-scale data transmission (such as DIVFusiontang2023divfusion, Ev-fusionzhang2024ev, L2fusiongao2023l2fusion, and LENFusionchen2024lenfusion). (b) The workflow of multi-scale data transmission (such as EFMNwang2023enlighten). Among them, the compared images are the results of DIVFusion (upper) and our method (lower). "E" denotes down-sampling, and "E" represents feature-expansion.
  • Figure 2: The overall architecture of our method. The parallel cascaded tasks include the infrared image-reconstruction task, illumination disentanglement task, and image fusion task. (a) The specific structure of the Details-Extraction Module, which aims to capture high-frequency features from the source images. (b) The architecture of the Hyper Cross-Attention Module, which is designed to obtain the low-frequency features.
  • Figure 3: (a) The visual results of iteration process in the Details-Extraction Module. (b)The interaction details of the Hyper Cross-Attention Module.
  • Figure 4: Vision quality comparison of five SOTA fusion methods on the LLVIP dataset. (a)-(b) Source images. (c) SeAFusiontang2022image. (d) U2Fusionxu2020u2fusion. (e) DenseFuseli2018densefuse. (f) TarDALliu2022target. (g) DFVO.
  • Figure 5: Human perception comparison of five SOTA fusion methods on the LLVIP dataset.
  • ...and 4 more figures