F-ViTA: Foundation Model Guided Visible to Thermal Translation

TL;DR

F-ViTA addresses the data-scarce challenge of visible-to-thermal translation by leveraging foundation models to guide a diffusion-based translator. It injects zero-shot object masks and labels from RAM-Grounded-SAM into an Instruct-Pix2Pix diffusion framework, training only the denoising U-Net and a projector to learn object-level heat correlations. Across five datasets spanning LWIR, MWIR, and NIR, F-ViTA outperforms state-of-the-art methods and demonstrates strong out-of-distribution generalization, plus the novel ability to generate LWIR/MWIR/NIR outputs from a single RGB image via text prompts. These capabilities open up practical paths for semi-supervised learning, RGB-T fusion, and downstream thermal segmentation, with future work exploring broader text-prompted inter-modality transfer.

Abstract

Thermal imaging is crucial for scene understanding, particularly in low-light and nighttime conditions. However, collecting large thermal datasets is costly and labor-intensive due to the specialized equipment required for infrared image capture. To address this challenge, researchers have explored visible-to-thermal image translation. Most existing methods rely on Generative Adversarial Networks (GANs) or Diffusion Models (DMs), treating the task as a style transfer problem. As a result, these approaches attempt to learn both the modality distribution shift and underlying physical principles from limited training data. In this paper, we propose F-ViTA, a novel approach that leverages the general world knowledge embedded in foundation models to guide the diffusion process for improved translation. Specifically, we condition an InstructPix2Pix Diffusion Model with zero-shot masks and labels from foundation models such as SAM and Grounded DINO. This allows the model to learn meaningful correlations between scene objects and their thermal signatures in infrared imagery. Extensive experiments on five public datasets demonstrate that F-ViTA outperforms state-of-the-art (SOTA) methods. Furthermore, our model generalizes well to out-of-distribution (OOD) scenarios and can generate Long-Wave Infrared (LWIR), Mid-Wave Infrared (MWIR), and Near-Infrared (NIR) translations from the same visible image. Code: https://github.com/JayParanjape/F-ViTA/tree/master.

Figures (5)

• Figure 1: Our model, F-ViTA, leverages pretrained foundation models to extract object tags and segmentation masks from visible images in a zero-shot manner, using this information to enhance translation to the thermal domain. Additionally, F-ViTA enables user-guided infrared image generation through text prompts, allowing for the synthesis of specific infrared types--an ability not explored in existing methods.
• Figure 2: An example usage of the RAM-Grounded-SAM pipeline. The dotted lines indicate the output at every step.
• Figure 3: Training pipeline. The stable diffusion part is responsible for learning the distribution of thermal images. The conditioning part provides the visible image to enforce structural similarity as well as guidance from the foundation model and text instructions for improved translation. Only the Denoising UNet and the Projector module are trainable in the pipeline.
• Figure 4: Qualitative Results over datasets from three different wavelength spectra. Our method (third column) is able to generate images more similar to the ground truth as compared to existing methods (fourth, fifth and sixth columns)
• Figure 5: Text prompted translation capability of F-ViTA. Our method is able to generate LWIR, MWIR or NIR images based on the text instruction. The second column shows the ground truth which is from the wavelength range specified next to the dataset name on the right.