Multi-Attribute guided Thermal Face Image Translation based on Latent Diffusion Model

TL;DR

This work tackles the problem of heterogeneous face recognition by translating thermal (thermal) images to visible-spectrum images while preserving identity features. It proposes a Latent Diffusion Model (LDM) framework conditioned on facial attributes, enhanced by a multi-attribute classifier, CLIP-based prompts, and the Self-Attn Mamba module to improve global cross-modal modeling and inference speed. Key contributions include (i) an attribute-conditioned LDM pipeline operating in a VQ-VAE latent space, (ii) a frozen yet trainable multi-attribute infrared classifier that aligns with visible attributes via CLIP prompts, and (iii) a Self-Attn Mamba module that reduces computational cost and improves identity preservation. Experimental results on ARL-VTF and SpeakingFaces datasets show state-of-the-art performance in both image quality and identity verification, with substantial speedups over prior diffusion-based approaches, making the method practical for real-time surveillance and forensic applications.

Abstract

Modern surveillance systems increasingly rely on multi-wavelength sensors and deep neural networks to recognize faces in infrared images captured at night. However, most facial recognition models are trained on visible light datasets, leading to substantial performance degradation on infrared inputs due to significant domain shifts. Early feature-based methods for infrared face recognition proved ineffective, prompting researchers to adopt generative approaches that convert infrared images into visible light images for improved recognition. This paradigm, known as Heterogeneous Face Recognition (HFR), faces challenges such as model and modality discrepancies, leading to distortion and feature loss in generated images. To address these limitations, this paper introduces a novel latent diffusion-based model designed to generate high-quality visible face images from thermal inputs while preserving critical identity features. A multi-attribute classifier is incorporated to extract key facial attributes from visible images, mitigating feature loss during infrared-to-visible image restoration. Additionally, we propose the Self-attn Mamba module, which enhances global modeling of cross-modal features and significantly improves inference speed. Experimental results on two benchmark datasets demonstrate the superiority of our approach, achieving state-of-the-art performance in both image quality and identity preservation.

Figures (6)

• Figure 1: Limitations of mainstream methods: GAN-based models often produce distorted or blurred images in T2V tasks, while diffusion-based methods struggle to accurately preserve facial features such as age, gender, and skin color.
• Figure 2: Pipeline of our method. Our model begins with training a VQ-VAE model from scratch, enabling the denoising process to operate in the latent space ($Z_t$ represents the latent features). We utilize a pre-trained classifier (parameters frozen, as shown in Figure \ref{['fig:classifer']}) to extract fine-grained facial attributes and skin tone labels from thermal images. These attributes are transformed into prompts using a customized CLIP encoder, guiding the generation process. Additionally, we integrate Self-attn mamba to further enhance inference speed and global feature modeling capabilities.
• Figure 3: Classifier module. During classifier training, an image pair consisting of an infrared image and an RGB face image $x$ is used as input. The RGB image $x_{vis}$ is processed by a frozen pre-trained recognition network, producing a feature vector $f_{vis}(x_{vis})$ at the max pooling layer after several MB modules. The infrared image $x_{IR}$ is processed by an untrained network with the same structure, producing a feature vector $f_{IR}(x_{IR})$. Consistency between $f_{IR}(x_{IR})$ and $f_{vis}(x_{vis})$ is enforced using MSE, defined as eq. \ref{['eq:MSE']} and multi-attribute classification is performed through a fully connected layer.
• Figure 4: Qualitative results on the ARL-VTF dataset for translating facial images from thermal to visible.
• Figure 5: Qualitative results on the SpeakingFaces dataset for translating facial images from thermal to visible.