Uniformity First: Uniformity-aware Test-time Adaptation of Vision-language Models against Image Corruption

TL;DR

This paper tackles the vulnerability of vision-language models, notably CLIP, to sensor degradation—a realistic distribution shift that standard test-time adaptation struggles with. It introduces UnInfo, a uniformity-aware, information-balanced TTA method that updates the image encoder with low-rank adapters (LoRA) and employs three components: uniformity-aware confidence maximization, adaptive loss balancing via mutual information, and EMA-based knowledge distillation. Empirical results on ImageNet-C and ImageNet-C-bar show that UnInfo consistently improves zero-shot accuracy across corruptions, with ablations demonstrating the necessity of each component and the benefit of balancing uniformity and entropy. The work advances robust deployment of CLIP-like models under real-world sensor degradations and suggests efficient online extensions as future work.

Abstract

Pre-trained vision-language models such as contrastive language-image pre-training (CLIP) have demonstrated a remarkable generalizability, which has enabled a wide range of applications represented by zero-shot classification. However, vision-language models still suffer when they face datasets with large gaps from training ones, i.e., distribution shifts. We found that CLIP is especially vulnerable to sensor degradation, a type of realistic distribution shift caused by sensor conditions such as weather, light, or noise. Collecting a new dataset from a test distribution for fine-tuning highly costs since sensor degradation occurs unexpectedly and has a range of variety. Thus, we investigate test-time adaptation (TTA) of zero-shot classification, which enables on-the-fly adaptation to the test distribution with unlabeled test data. Existing TTA methods for CLIP mainly focus on modifying image and text embeddings or predictions to address distribution shifts. Although these methods can adapt to domain shifts, such as fine-grained labels spaces or different renditions in input images, they fail to adapt to distribution shifts caused by sensor degradation. We found that this is because image embeddings are "corrupted" in terms of uniformity, a measure related to the amount of information. To make models robust to sensor degradation, we propose a novel method called uniformity-aware information-balanced TTA (UnInfo). To address the corruption of image embeddings, we introduce uniformity-aware confidence maximization, information-aware loss balancing, and knowledge distillation from the exponential moving average (EMA) teacher. Through experiments, we demonstrate that our UnInfo improves accuracy under sensor degradation by retaining information in terms of uniformity.

Figures (4)

• Figure 1: Overview of Uniformity-aware Information-balanced Test-time Adaptation (UnInfo).
• Figure 2: Sensitivity analysis on the weight of the uniformity loss $\lambda$ (top) and threshold of the mutual information used in the loss balancing $\mathcal{I}_0$ (bottom). The mean and standard deviation of the test accuracy calculated over the ImageNet-C corruptions are plotted.
• Figure 3: Evolution of the information-aware loss balancing weights. The weights are adaptively assigned to the entropy and uniformity losses by the difficulty of distribution shifts. A larger weight is assigned to the entropy for easy distribution shifts such as brightness in (a). In contrast, the uniformity loss is first addressed, and then the weight is gradually switched to the entropy for difficult distribution shifts such as Gaussian noise in (d).
• Figure 4: Spherical PCA liu2019spherical visualization of image (blue dots) and text (orange dots) embeddings before and after TTA with UnInfo. The image embeddings are distributed in a broader range of the circle after TTA, which suggests that the uniformity is improved. Moreover, the distribution of the image embeddings is aligned with the text embeddings, i.e., the image embeddings are more classification-friendly.