Data Generation Scheme for Thermal Modality with Edge-Guided Adversarial Conditional Diffusion Model

TL;DR

This work addresses the data scarcity of thermal imagery for robust object detection by proposing an edge-guided conditional diffusion framework (ECDM) that generates pixel-level aligned thermal images from visible-edge cues. A two-stage modality adversarial training (TMAT) is introduced to mitigate leakage of unwanted visible-edge information, enabling learning of $p_{model}(x^{tir}| ext{edge})$ and improving cross-domain transfer. Empirical results on LLVIP, FLIR, and PRW demonstrate superior image-quality metrics and up to 7.1 mAP gains across detectors when using ECDM-generated pseudo-thermal data, with notable improvements in transferability and edge-constraint fidelity. The approach offers a scalable data-generation pathway to bolster thermal vision in low-light and adverse conditions, with potential extensions to other cross-domain synthesis tasks.

Abstract

In challenging low light and adverse weather conditions,thermal vision algorithms,especially object detection,have exhibited remarkable potential,contrasting with the frequent struggles encountered by visible vision algorithms. Nevertheless,the efficacy of thermal vision algorithms driven by deep learning models remains constrained by the paucity of available training data samples. To this end,this paper introduces a novel approach termed the edge guided conditional diffusion model. This framework aims to produce meticulously aligned pseudo thermal images at the pixel level,leveraging edge information extracted from visible images. By utilizing edges as contextual cues from the visible domain,the diffusion model achieves meticulous control over the delineation of objects within the generated images. To alleviate the impacts of those visible-specific edge information that should not appear in the thermal domain,a two-stage modality adversarial training strategy is proposed to filter them out from the generated images by differentiating the visible and thermal modality. Extensive experiments on LLVIP demonstrate ECDM s superiority over existing state-of-the-art approaches in terms of image generation quality.

Figures (5)

• Figure 1: A sample comparison of generated thermal images between different methods and ground truth. (\ref{['Fig:gt']}) A ground truth thermal image, (\ref{['Fig:ugatit']}) a generated thermal image by UGATIT, (\ref{['Fig:ddim']}) a generated thermal image by DDIM and (\ref{['Fig:ecdm']}) a generated thermal image by ECDM (Ours).
• Figure 2: The performance of RetinaNet trained with various amounts of generated pseudo training data. The x-axis indicates the augmentation multiple. For example, 0.2 indicates that the generated pseudo training data in the entire training sample is only 20% of the real data.
• Figure 3: Illustration of our Two-stage Modality Adversarial Training (TMAT) strategy. During the first stage, we only use $x^{tir}$ as input and train the ECDM to learn the distribution of $p_{model}(x^{tir}|\zeta ^{tir})$. In the second stage, we use unpaired $x^{vis}$ and $x^{tir}$ as input and utilize GANs to reduce the gap between visible and thermal domains. This helps us learn the distribution of $p_{model}(x^{tir}|\zeta ^{vis})$ for approximating $p_{tir}(x^{tir})$.
• Figure 4: The architecture of ECDM. The numbers over the light blue rectangle blocks denote the channels of feature maps. The yellow rectangle block denotes an attention layer.
• Figure 5: Qualitative comparison of our proposed method with other state-of-the-art methods on the LLVIP test dataset. To ensure fairness and randomness, we use Python's random module with a fixed seed (1234) to select four images from the dataset. The selected images are '190145.jpg', '190345.jpg', '190373.jpg', '190405.jpg', '190480.jpg', '220224.jpg', '260261.jpg'. More visual results can be found in supplementary material.