CLIPArTT: Adaptation of CLIP to New Domains at Test Time

TL;DR

CLIPArTT addresses the vulnerability of vision-language models like CLIP to domain shifts by introducing a fully test-time adaptation approach that generates instance-specific text prompts from top-K predictions and updates only normalization layers. It introduces a transductive TTA loss that leverages batch-aware image-image and text-text similarities to produce soft pseudo-labels, improving robustness without requiring model retraining. The method demonstrates strong, consistent gains across natural and corrupted datasets (e.g., CIFAR-10/10-C/100, CIFAR-100-C, ImageNet-C, VisDA-C) and establishes a standardized TTA benchmark for vision-language models. The work highlights the practicality of lightweight, prompt-informed test-time adaptation and points to future extensions in segmentation, open-set handling, and prompt design.

Abstract

Pre-trained vision-language models (VLMs), exemplified by CLIP, demonstrate remarkable adaptability across zero-shot classification tasks without additional training. However, their performance diminishes in the presence of domain shifts. In this study, we introduce CLIP Adaptation duRing Test-Time (CLIPArTT), a fully test-time adaptation (TTA) approach for CLIP, which involves automatic text prompts construction during inference for their use as text supervision. Our method employs a unique, minimally invasive text prompt tuning process, wherein multiple predicted classes are aggregated into a single new text prompt, used as \emph{pseudo label} to re-classify inputs in a transductive manner. Additionally, we pioneer the standardization of TTA benchmarks (e.g., TENT) in the realm of VLMs. Our findings demonstrate that, without requiring additional transformations nor new trainable modules, CLIPArTT enhances performance dynamically across non-corrupted datasets such as CIFAR-100, corrupted datasets like CIFAR-100-C and ImageNet-C, alongside synthetic datasets such as VisDA-C. This research underscores the potential for improving VLMs' adaptability through novel test-time strategies, offering insights for robust performance across varied datasets and environments. The code can be found at: https://github.com/dosowiechi/CLIPArTT.git

Figures (4)

• Figure 1: CLIPArTT pipeline overview: 1) Computing predictions from Image-Text Similarity, 2) generating a new text prompt by filtering the top-$K$ class predictions, 3) with the new prompts, a pseudo-label $\textbf{Q}$ is obtained by averaging the image-to-image and text-to-text similarity scores, while the prediction $\hat{\textbf{P}}$ is computed as the image-to-text similarity. Cross-entropy is then used as the TTA loss.
• Figure 2: Top: Example of similarity matrices ($\mathbf{S}^v$, $\mathbf{S}^t$) and CLIPArTT softmax probabilities ($\mathbf{Q}$) for a batch of 5 examples and using $K=5$ classes. Bottom: a) When using the identity matrix as pseudo-label for contrastive learning, the correct prediction is ambiguous, as the images are forced to both approaching and moving away from the right class. b) CLIPArTT uses soft pseudo-labels that smoothly guides the prediction towards the correct class by reducing the impact of ambiguities in the prompts.
• Figure 3: Evolution of CLIPArTT's accuracy during test-time adaptation. Left: For different versions of CIFAR10. Right: Compared to TENT on CIFAR10-C.
• Figure 4: The t-SNE visualizations exhibit discernible attributes of brightness within the visual features derived from CLIPArTT. Panels (a) and (b) present the model's predictions before and after 10 iterations of adaptation, respectively. Panels (c) and (d) demonstrate the actual labels in the absence of adaptation and following adaptation of the representations, respectively.

Theorems & Definitions (2)

• Proposition 1
• proof