Diffusion-Driven Progressive Target Manipulation for Source-Free Domain Adaptation

TL;DR

This work tackles source-free domain adaptation by directly generating a pseudo-target domain rather than a pseudo-source, addressing large source–target gaps with a diffusion-driven, progressive refinement strategy. By partitioning unlabeled target data into a trust set with reliable pseudo-labels and a non-trust set that is semantically transformed toward assigned labels using Target-guided Initialization, Semantic Feature Injection, and Domain-specific Feature Preservation, DPTM reduces domain discrepancy iteratively over $R$ refinements. Empirical results across Office-31, Office-Home, VisDA, and DomainNet-126 demonstrate state-of-the-art performance, with notable gains in challenging transfer tasks and scenarios with substantial domain shifts. The approach combines diffusion-based manipulation with a structured refinement loop to robustly align pseudo-target data with the real target distribution, offering practical gains for SFDA in real-world deployments.

Abstract

Source-free domain adaptation (SFDA) is a challenging task that tackles domain shifts using only a pre-trained source model and unlabeled target data. Existing SFDA methods are restricted by the fundamental limitation of source-target domain discrepancy. Non-generation SFDA methods suffer from unreliable pseudo-labels in challenging scenarios with large domain discrepancies, while generation-based SFDA methods are evidently degraded due to enlarged domain discrepancies in creating pseudo-source data. To address this limitation, we propose a novel generation-based framework named Diffusion-Driven Progressive Target Manipulation (DPTM) that leverages unlabeled target data as references to reliably generate and progressively refine a pseudo-target domain for SFDA. Specifically, we divide the target samples into a trust set and a non-trust set based on the reliability of pseudo-labels to sufficiently and reliably exploit their information. For samples from the non-trust set, we develop a manipulation strategy to semantically transform them into the newly assigned categories, while simultaneously maintaining them in the target distribution via a latent diffusion model. Furthermore, we design a progressive refinement mechanism that progressively reduces the domain discrepancy between the pseudo-target domain and the real target domain via iterative refinement. Experimental results demonstrate that DPTM outperforms existing methods by a large margin and achieves state-of-the-art performance on four prevailing SFDA benchmark datasets with different scales. Remarkably, DPTM can significantly enhance the performance by up to 18.6% in scenarios with large source-target gaps.

Figures (5)

• Figure 1: In DPTM, we employ progressive refinement $R$ times: First, we use the target model to make predictions on the target data. Based on each sample's prediction uncertainty, we divide the target data into a trust set and a non-trust set. For the low-uncertainty trust set, we train the target model using pseudo-labels in a supervised manner. For the high-uncertainty non-trust set, we assign a label $\hat{y}_l$ for each sample $\mathbf{x}_l^u$, employ a manipulation strategy that semantically transforms $\mathbf{x}_l^u$ toward class $\hat{y}_l$, while preserving the target-domain features of $\mathbf{x}_l^u$. Our manipulation consists of three components: Target-guided Initialization to obtain an effective sampling starting point, Semantic Feature Injection to convert the semantics of the generated sample to $\hat{y}_l$, and Domain-specific Feature Preservation to maintain the generated sample within the target distribution.
• Figure 2: Ablation on Manipulation Mechanism of $\mathbf{x}_l^u$. Row: $\hat{y}_l=$ 'Alarm Clock', 'Curtains', 'Computer', 'Bottle', respectively. Column: (a) $\mathbf{x}_l^u$ (b) $\tilde{\mathbf{x}}_l^{u}$ w/o Target-guided Initialization (c) $\tilde{\mathbf{x}}_l^{u}$ w/o Semantic Feature Injection (d) $\tilde{\mathbf{x}}_l^{u}$ w/o Domain-specific Feature Preservation (e) $\tilde{\mathbf{x}}_l^{u}$ of our method.
• Figure 3: The relationship between $r$ versus: (a) The number of samples in the trust set. (b) The trust set accuracy. (c) The number of samples in the non-trust set.
• Figure A.1: Feature distribution visualization on the Rw$\to$Cl task of the Office-Home dataset.
• Figure A.2: Grad-CAM visualization on the Office-Home dataset.