DiffV2IR: Visible-to-Infrared Diffusion Model via Vision-Language Understanding

TL;DR

DiffV2IR addresses the challenging problem of translating visible images to infrared while preserving semantic content and handling the wide variability of infrared spectra. It harmonizes a Progressive Learning Module (PLM) that incrementally equips a diffusion model with infrared knowledge and cross-modal translation capabilities with a Vision-Language Understanding Module (VLUM) that injects detailed semantic and structural cues via vision-language embeddings and segmentation maps. The large IR-500K dataset, together with ~70K visible–infrared pairs, enables robust multi-stage training and style-controllable outputs. Empirical results on M$^3$FD and FLIR-aligned datasets show state-of-the-art performance in FID, PSNR, and SSIM, validating DiffV2IR's semantic-aware, cross-modal translation and its potential for broad multi-spectral image generation applications.

Abstract

The task of translating visible-to-infrared images (V2IR) is inherently challenging due to three main obstacles: 1) achieving semantic-aware translation, 2) managing the diverse wavelength spectrum in infrared imagery, and 3) the scarcity of comprehensive infrared datasets. Current leading methods tend to treat V2IR as a conventional image-to-image synthesis challenge, often overlooking these specific issues. To address this, we introduce DiffV2IR, a novel framework for image translation comprising two key elements: a Progressive Learning Module (PLM) and a Vision-Language Understanding Module (VLUM). PLM features an adaptive diffusion model architecture that leverages multi-stage knowledge learning to infrared transition from full-range to target wavelength. To improve V2IR translation, VLUM incorporates unified Vision-Language Understanding. We also collected a large infrared dataset, IR-500K, which includes 500,000 infrared images compiled by various scenes and objects under various environmental conditions. Through the combination of PLM, VLUM, and the extensive IR-500K dataset, DiffV2IR markedly improves the performance of V2IR. Experiments validate DiffV2IR's excellence in producing high-quality translations, establishing its efficacy and broad applicability. The code, dataset, and DiffV2IR model will be available at https://github.com/LidongWang-26/DiffV2IR.

Figures (6)

• Figure 1: Main challenges of V2IR. (a) Semantic-aware translation, in which the context information of shadow influences the infrared image a lot. (b) Diverse infrared radiations. Even similar visual scenes from different infrared cameras show the diversity of infrared imagery. For the second column to the fourth column, the infrared intensity significantly changes.
• Figure 2: Framework overview of our DiffV2IR. DiffV2IR mainly consists of two components, i.e., Progressive Learning Module (PLM) and Vision-Language Understanding Module (VLUM). We use PLM for multi-stage knowledge learning and VLUM for semantic preserving in the V2IR task. The three U-Nets from bottom to top respectively denote the infrared representation internalization phase, the cross-modal transformation learning phase, and stylization refinement phase of PLM. The VLUM is introduced during PLM to make DiffV2IR semantic-aware.
• Figure 3: Comparison with SOTA methods. Key differences are highlighted within a red box, such as halos and low-light scenarios. Only the top 5 methods according to assessment metrics are shown. (Top: results from M$^3$FD dataset; Bottom: results from FLIR-aligned dataset.)
• Figure I: Intermediate results of our proposed DiffV2IR.
• Figure II: Comparison with SOTA methods on M$^3$FD dataset.