AD-CLIP: Adapting Domains in Prompt Space Using CLIP

TL;DR

AD-CLIP is introduced, a domain-agnostic prompt learning strategy for CLIP that aims to solve the DA problem in the prompt space, and uses the frozen vision backbone of CLIP to extract both image style and content information, which is applied to learn prompt tokens.

Abstract

Although deep learning models have shown impressive performance on supervised learning tasks, they often struggle to generalize well when the training (source) and test (target) domains differ. Unsupervised domain adaptation (DA) has emerged as a popular solution to this problem. However, current DA techniques rely on visual backbones, which may lack semantic richness. Despite the potential of large-scale vision-language foundation models like CLIP, their effectiveness for DA has yet to be fully explored. To address this gap, we introduce \textsc{AD-CLIP}, a domain-agnostic prompt learning strategy for CLIP that aims to solve the DA problem in the prompt space. We leverage the frozen vision backbone of CLIP to extract both image style (domain) and content information, which we apply to learn prompt tokens. Our prompts are designed to be domain-invariant and class-generalizable, by conditioning prompt learning on image style and content features simultaneously. We use standard supervised contrastive learning in the source domain, while proposing an entropy minimization strategy to align domains in the embedding space given the target domain data. We also consider a scenario where only target domain samples are available during testing, without any source domain data, and propose a cross-domain style mapping network to hallucinate domain-agnostic tokens. Our extensive experiments on three benchmark DA datasets demonstrate the effectiveness of \textsc{AD-CLIP} compared to existing literature. Code is available at \url{https://github.com/mainaksingha01/AD-CLIP}

Figures (8)

• Figure 1: We compare the performance of AD-CLIP for the Office-Home officehome dataset, with different type of UDA methods, e.g. convnets-based DANN dann, Transformer-based CDTrans cdtrans, pre-trained CLIP clip without prompt learning and, DAPL dapl, a prompt learning-based DA technique.
• Figure 2: We highlight the differences between our prompts from the literature. a) CoOp coop directly learns the prompt tokens from random vectors and may not be suitable for DA as it does not concern domain independence, b) Another possibility is to manually include the domain name into manually defined prompts, but this information may not be readily available, c) AD-CLIP introduces an automatic solution by leveraging the visual space to define the domain-agnostic and image-conditioned tokens.
• Figure 3: The architecture of AD-CLIP is based on the frozen CLIP backbones $f_v$ and $f_t$. For prompt token learning, we introduce the new vision and text projectors $P_v$ and $C_v$, respectively, which encoder the style and content information from the different layers of $f_v$. The style mapping network, $P_{smn}$, approximates the source domain style information from the target domain features. Best viewed in color.
• Figure 4: t-SNE visualizations of text embeddings from art and clipart domains of 10 classes of Office-Home.
• Figure 5: Performance of AD-CLIP with different layers of RN50, ViT-B/16 and ViT-L/14 backbones to extract multi-scale features on Office-Home.