CLIP the Divergence: Language-guided Unsupervised Domain Adaptation

TL;DR

This work introduces CLIP-Div, a language-guided unsupervised domain adaptation method that directly harnesses CLIP to derive a domain-agnostic distribution and guide alignment between source and target domains. It defines two novel divergence losses—absolute and relative—together with a language-guided pseudo-labeling strategy to calibrate target labels and enable self-training. By freezing CLIP encoders and using domain-specific prompts alongside a domain-agnostic prompt, the approach achieves state-of-the-art performance across Office-Home, Office-31, VisDA-2017, and DomainNet, with substantial gains over CNN-based methods. The results underscore the value of language guidance in bridging domain gaps and refining pseudo-labels for robust target-domain generalization, while acknowledging limitations tied to fixed CLIP representations and the potential for further gains with prompt learning or fine-tuning in future work.

Abstract

Unsupervised domain adaption (UDA) has emerged as a popular solution to tackle the divergence between the labeled source and unlabeled target domains. Recently, some research efforts have been made to leverage large vision-language models, such as CLIP, and then fine-tune or learn prompts from them for addressing the challenging UDA task. In this work, we shift the gear to a new direction by directly leveraging CLIP to measure the domain divergence and propose a novel language-guided approach for UDA, dubbed as CLIP-Div. Our key idea is to harness CLIP to 1) measure the domain divergence via the acquired domain-agnostic distribution and 2) calibrate the target pseudo labels with language guidance, to effectively reduce the domain gap and improve the UDA model's generalization capability. Specifically, our major technical contribution lies in the proposed two novel language-guided domain divergence measurement losses: absolute divergence and relative divergence. These loss terms furnish precise guidelines for aligning the distributions of the source and target domains with the domain-agnostic distribution derived from CLIP. Additionally, we propose a language-guided pseudo-labeling strategy for calibrating the target pseudo labels. Buttressed by it, we show that a further implementation for self-training can enhance the UDA model's generalization capability on the target domain. CLIP-Div surpasses state-of-the-art CNN-based methods by a substantial margin, achieving a performance boost of +10.3% on Office-Home, +1.5% on Office-31, +0.2% on VisDA-2017, and +24.3% on DomainNet, respectively.

Figures (3)

• Figure 1: The key intuition of our CLIP-Div for measuring the domain divergence. The domain-agnostic distribution, acquired through CLIP with language guidance, serves as a pivotal bridge between source and target domains through the design of divergence measurements.
• Figure 2: (Left) Overview of the pipeline of our proposed CLIP-Div. The learned UDA model comprises a feature extractor $\mathcal{F}$ and a classifier $\mathcal{G}$. The encoders of frozen CLIP, denoted as $\mathcal{E_I}$ and $\mathcal{E_T}$, are harnessed for the acquisition of a domain-agnostic distribution. (Right) The intuition of our two proposed language-guided divergence measurement losses and their collaborative effects. (i) In UDA, the objective is to align source and target domains by learning domain-invariant representations; (ii) the absolute divergence measurement loss works to bring the distributions of both domains closer to the domain-agnostic distribution; (iii) the relative divergence measurement loss facilitates the domain alignment based on the distance between samples; (iv) collectively, these language-guided divergence measurement losses contribute to mitigating the domain gap between the source and target domains.
• Figure 3: The t-SNE visualization with the proposed losses on the task A $\to$ P.