BlenDA: Domain Adaptive Object Detection through diffusion-based blending

TL;DR

BlenDA tackles unsupervised domain adaptation for object detection by generating intermediate-domain samples through diffusion-based blending of source and target-like translated images. It introduces dynamic mixing with a soft domain label for adversarial learning, enabling finer-grained domain alignment within an Adversarial Query Transformer framework. The method achieves substantial gains on Cityscapes→Foggy Cityscapes (up to 53.4% mAP) and Cityscapes→BDD100K daytime (≈+4.1% mAP), and ablations confirm the benefits of dynamic blending and soft-domain losses. This approach offers a versatile, diffusion-assisted bridge between domains and can be integrated with various domain-adaptive detectors, providing a practical improvement for real-world cross-domain object detection tasks.

Abstract

Unsupervised domain adaptation (UDA) aims to transfer a model learned using labeled data from the source domain to unlabeled data in the target domain. To address the large domain gap issue between the source and target domains, we propose a novel regularization method for domain adaptive object detection, BlenDA, by generating the pseudo samples of the intermediate domains and their corresponding soft domain labels for adaptation training. The intermediate samples are generated by dynamically blending the source images with their corresponding translated images using an off-the-shelf pre-trained text-to-image diffusion model which takes the text label of the target domain as input and has demonstrated superior image-to-image translation quality. Based on experimental results from two adaptation benchmarks, our proposed approach can significantly enhance the performance of the state-of-the-art domain adaptive object detector, Adversarial Query Transformer (AQT). Particularly, in the Cityscapes to Foggy Cityscapes adaptation, we achieve an impressive 53.4% mAP on the Foggy Cityscapes dataset, surpassing the previous state-of-the-art by 1.5%. It is worth noting that our proposed method is also applicable to various paradigms of domain adaptive object detection. The code is available at:https://github.com/aiiu-lab/BlenDA

Figures (3)

• Figure 1: Overview of BlenDA. We utilize a diffusion-based generative model Instruction to generate a target-like translated image with an appropriate prompt and mix it with the source image into a series of blended images for the intermediate domains, which play the role of a bridge between two domains and can be used to train a detector progressively. The direction of the arrow (Blue one) represents an increasing proportion of the translated image used to blend with the source image.
• Figure 2: Comprehensive overview of BlenDA on AQT AQT. We first generate translated images by InstructPix2Pix Instruction and compute the $\delta$ using Eq \ref{['eqn:delta']}. Subsequently, we feed the images $I_{blended}$ and $I_{s,t}$, which are mixed according to Eq \ref{['fix_delta']}, into the model. $I_{blended}$ replaces the original source image during the fine-tuning process, using the supervised loss $L_{sup}$ to reduce the domain gap. Moreover, we adjust the original adversarial loss specified in Eq \ref{['eqn:L_adv']} to Eq \ref{['eqn:L_adv_mixed']}. This new adversarial loss enables the discriminators to distinguish different domains in greater detail. Note that the model weights are initialized using the pre-trained weights released by Huang et al. AQT before fine-tuning.
• Figure 3: Cityscapes Cityscapes images are fed into InstructPix2Pix Instruction with the prompts to create translated images for both Foggy Cityscapes Foggy_Cityscapes and BDD100K daytime BDD_daytime. Foreground objects are notably absent, as indicated by the red boxes in the translated images. After mixing the source and translated images, foreground objects become visible as outlined by blue boxes in the blended images. $\delta$ represents the proportion used to blend the translated image with the source one.