Mitigating Cache Noise in Test-Time Adaptation for Large Vision-Language Models

TL;DR

This work tackles cache-based test-time adaptation for large vision-language models under distribution shift, where noisy pseudo-labels undermine cache quality. It proposes Cache, Residual, Gaussian (CRG), a zero-shot TTA framework that maintains separate positive and negative image caches plus a text prototype cache, coupled with learnable residuals to calibrate multi-modal prototypes. Gaussian Discriminant Analysis is employed to model intra-class distributions and mitigate the impact of noisy features, while a negative prototype mechanism further reduces overconfidence. Across 13 benchmarks with cross-dataset and natural distribution shifts, CRG delivers state-of-the-art robustness and adaptability, demonstrating the practical viability of distribution-aware, cache-based TTA for CLIP-like models.

Abstract

Test-time adaptation (TTA) of visual language models has recently attracted significant attention as a solution to the performance degradation caused by distribution shifts in downstream tasks. However, existing cache-based TTA methods have certain limitations. They mainly rely on the accuracy of cached feature labels, and the presence of noisy pseudo-labels can cause these features to deviate from their true distribution. This makes cache retrieval methods based on similarity matching highly sensitive to outliers or extreme samples. Moreover, current methods lack effective mechanisms to model class distributions, which limits their ability to fully exploit the potential of cached information. To address these challenges, we introduce a comprehensive and reliable caching mechanism and propose a novel zero-shot TTA method called "Cache, Residual, Gaussian" (CRG). This method not only employs learnable residual parameters to better align positive and negative visual prototypes with text prototypes, thereby optimizing the quality of cached features, but also incorporates Gaussian Discriminant Analysis (GDA) to dynamically model intra-class feature distributions, further mitigating the impact of noisy features. Experimental results on 13 benchmarks demonstrate that CRG outperforms state-of-the-art TTA methods, showcasing exceptional robustness and adaptability.

Figures (3)

• Figure 1: Figure (a) shows the change in error rates for the positive cache during downstream testing with the TDATDA method, while Figure (b) compares the performance gap between similarity-based cache models in zero-shot(TDATDA) and few-shot (Tip-adapterTipAdapter) settings, all using RN50 as the backbone.
• Figure 2: Overview of the CRG Method. We introduce caches (both positive and negative) in the text and visual modalities and employ learnable residual vectors to flexibly calibrate multi-modal features. During inference, positive prototypes, negative prototypes, and text prototypes are used for similarity matching, while GDA models the feature distribution to mitigate noise interference in predictions. By simultaneously minimizing prediction entropy and maximizing inter-prototype distances, CRG achieves multi-modal alignment with enhanced robustness and generalization.
• Figure 3: Two classic methods of Test-Time Adaptation (top) and our cache-based approach (bottom).