Source-Free Domain Adaptation Guided by Vision and Vision-Language Pre-Training

TL;DR

This work tackles the problem of source-free domain adaptation by revealing that target pseudolabels from a solely source-trained model can be biased under domain shift. It proposes a flexible two-branch framework, Co-learn, that retains a pre-trained network alongside the adaptation model to distill target information during target adaptation, and extends it with Co-learn++ that incorporates CLIP's zero-shot decisions. Through extensive experiments on Office-31, Office-Home, VisDA-C, and DomainNet, the approach improves target performance and robustly handles open-set and partial-set SFDA variants, often outperforming existing SFDA methods. The findings highlight the practical value of leveraging modern pre-trained vision and vision-language models to enhance SFDA without additional data or training on the source domain.

Abstract

Source-free domain adaptation (SFDA) aims to adapt a source model trained on a fully-labeled source domain to a related but unlabeled target domain. While the source model is a key avenue for acquiring target pseudolabels, the generated pseudolabels may exhibit source bias. In the conventional SFDA pipeline, a large data (e.g. ImageNet) pre-trained feature extractor is used to initialize the source model at the start of source training, and subsequently discarded. Despite having diverse features important for generalization, the pre-trained feature extractor can overfit to the source data distribution during source training and forget relevant target domain knowledge. Rather than discarding this valuable knowledge, we introduce an integrated framework to incorporate pre-trained networks into the target adaptation process. The proposed framework is flexible and allows us to plug modern pre-trained networks into the adaptation process to leverage their stronger representation learning capabilities. For adaptation, we propose the Co-learn algorithm to improve target pseudolabel quality collaboratively through the source model and a pre-trained feature extractor. Building on the recent success of the vision-language model CLIP in zero-shot image recognition, we present an extension Co-learn++ to further incorporate CLIP's zero-shot classification decisions. We evaluate on 4 benchmark datasets and include more challenging scenarios such as open-set, partial-set and open-partial SFDA. Experimental results demonstrate that our proposed strategy improves adaptation performance and can be successfully integrated with existing SFDA methods. Project code is available at https://github.com/zwenyu/colearn-plus.

Figures (8)

• Figure 1: VisDA-C source-trained ResNet-101 produces unreliable pseudolabels on target samples, and is over-confident on a significant number of incorrect predictions.
• Figure 2: Overview of conventional and proposed framework. We propose incorporating pre-trained networks during target adaptation. For the pre-trained network, we can plug in the same network used for source model initialization, or a different network that potentially has better feature extraction capabilities on the target domain.
• Figure 3: Overview of proposed strategy: (i) Source model trained on source domain is provided. (ii) We adapt the source model through a co-learning strategy where the adaptation model $\{f_a, k_a\}$ and a large data pre-trained feature extractor $f_*$ collectively produce more reliable pseudolabels for finetuning. To estimate a new classification head on $f_*$, the Co-learn algorithm computes a nearest-centroid-classifier $q_*$ weighted by adaptation model predictions, and the Co-learn++ algorithm additionally integrates CLIP zero-shot predictions from the text-based classifier $\{\bar{f}_a, \bar{k}_a\}$ to compute $q^{\texttt{++}}_*$.
• Figure 4: t-SNE visualization of VisDA-C target domain features by (a) ImageNet-1k ResNet-101, (b) source-trained ResNet-101, (c) ImageNet-1k Swin-B, (d) source ResNet-101 adapted by co-learning with ImageNet-1k Swin-B, (e) WIT CLIP, and (f) source ResNet-101 adapted by co-learning with WIT CLIP. Features are extracted at the last pooling layer before the classification head. Samples are colored by class.
• Figure 5: VisDA-C co-learning training curves, with ImageNet-1k ConvNeXt-S in pre-trained model branch.