Domain Adaptation via Prompt Learning

TL;DR

The paper tackles unsupervised domain adaptation by moving away from feature alignment and toward per-domain prompts within a vision-language framework (CLIP).It introduces Domain Adaptation via Prompt Learning (DAPL), which uses domain-agnostic and domain-specific prompts and leverages a contrastive objective plus pseudo-labeling to align image and text representations across domains.Empirical results on Office-Home and VisDA-2017 show state-of-the-art performance and notable training efficiency, underscoring the effectiveness of prompt-based domain conditioning.This work contributes a novel integration of prompt learning with UDA, preserving semantic discriminability under domain shift and enabling scalable adaptation with minimal parameter updates.

Abstract

Unsupervised domain adaption (UDA) aims to adapt models learned from a well-annotated source domain to a target domain, where only unlabeled samples are given. Current UDA approaches learn domain-invariant features by aligning source and target feature spaces. Such alignments are imposed by constraints such as statistical discrepancy minimization or adversarial training. However, these constraints could lead to the distortion of semantic feature structures and loss of class discriminability. In this paper, we introduce a novel prompt learning paradigm for UDA, named Domain Adaptation via Prompt Learning (DAPL). In contrast to prior works, our approach makes use of pre-trained vision-language models and optimizes only very few parameters. The main idea is to embed domain information into prompts, a form of representations generated from natural language, which is then used to perform classification. This domain information is shared only by images from the same domain, thereby dynamically adapting the classifier according to each domain. By adopting this paradigm, we show that our model not only outperforms previous methods on several cross-domain benchmarks but also is very efficient to train and easy to implement.

Figures (5)

• Figure 1: Overview of DAPL. We introduce the prompt tuning framework for domain adaptation. Top: conventional domain adaptation methods aim to remove domain-specific information via domain alignment or adversarial loss. This could lead to distorted feature representation when the manifold structures underlying the data distributions are complex cai2019learning. Bottom: Our method preserves domain information and tunes a prompt for each domain. Our model learns with a contrastive objective.
• Figure 2: Example prompt structure. Our proposed prompt consists of three parts: (a) Domain-specific prompt; (b) Domain-agnostic prompt; (c) Class label. The first two parts are continuous and learned from data. The words shown here are for illustrative purposes.
• Figure 3: Domain Adaptation via Prompt Learning (DAPL): (a) DAPL trains the learnable context variables: domain-agnostic context variables and domain-specific context variables, and [CLASS] token which are combined and encoded by a text encoder. (b) An image encoder encodes images from different domains. (c) Next, cosine similarity between text and image features is computed and the positive pairs (with matched domain and class) are encouraged to align. The classification probability are defined in \ref{['eq:pro']} and a cross-entropy loss is applied between the image feature and the ground truth class to train the networks.
• Figure 4: Contrastive learning helps transfer learning. (a) We assume that visual representation implicitly contains two parts: domain information ($\mathbf{z}_d$) and class information ($\mathbf{z}_c$). Similarly, the language feature contains two parts: domain information ($\mathbf{p}_d$) and class information ($\mathbf{p}_c$). By minimizing the distance between positive pairs (shown in green) and maximizing the distance between negative pairs (shown in red), we show that the domain information and class information can be disentangled. Such disentangled representations can be applied for transfer learning. See \ref{['sec:disentangle']} for details.
• Figure 5: Prediction confidence from VisDA-2017 (top) and Office-Home dataset (bottom). Confidence of the ground-truth class predicted using different prompting methods. Blue: manually designed prompt. Green: domain-agnostic prompt. Pink: our proposed method. Predictions given by our method show the highest confidence.