Class-Aware Prototype Learning with Negative Contrast for Test-Time Adaptation of Vision-Language Models

TL;DR

CPL-NC tackles the key challenges of test-time adaptation for Vision-Language Models: prototype forgetting in long-tailed distributions and confusion among semantically similar classes. It introduces a Class-Aware Prototype Cache that dynamically allocates per-class memory with a rejuvenation mechanism for inactive tail classes, and a Negative Contrastive Learning module that sharpens class boundaries by using hard cross-modal negatives. The framework employs asymmetric optimization, updating textual prototypes while keeping visual prototypes anchored, and fuses memory with refined prompts during inference. Extensive experiments on 15 benchmarks show CPL-NC consistently improves robustness and generalization across both ResNet-50 and ViT-B/16 backbones, often surpassing prior TTA methods with a favorable efficiency–accuracy trade-off.

Abstract

Vision-Language Models (VLMs) demonstrate impressive zero-shot generalization through large-scale image-text pretraining, yet their performance can drop once the deployment distribution diverges from the training distribution. To address this, Test-Time Adaptation (TTA) methods update models using unlabeled target data. However, existing approaches often ignore two key challenges: prototype degradation in long-tailed distributions and confusion between semantically similar classes. To tackle these issues, we propose \textbf{C}lass-Aware \textbf{P}rototype \textbf{L}earning with \textbf{N}egative \textbf{C}ontrast(\textbf{CPL-NC}), a lightweight TTA framework designed specifically for VLMs to enhance generalization under distribution shifts. CPL-NC introduces a \textit{Class-Aware Prototype Cache} Module that dynamically adjusts per-class capacity based on test-time frequency and activation history, with a rejuvenation mechanism for inactive classes to retain rare-category knowledge. Additionally, a \textit{Negative Contrastive Learning} Mechanism identifies and constrains hard visual-textual negatives to improve class separability. The framework employs asymmetric optimization, refining only textual prototypes while anchoring on stable visual features. Experiments on 15 benchmarks show that CPL-NC consistently outperforms prior TTA methods across both ResNet-50 and ViT-B/16 backbones.

Figures (3)

• Figure 1: Comparative results of CPL-NC on (a) OOD robustness using ViT-B/16 and (b) cross-dataset generalization using ResNet-50.(c) Sample distribution of the top-5 and bottom-5 categories in ImageNet-A hendrycks2021natural, illustrating significant class imbalance. Some minority classes account for less than 1% of total samples, while dominant classes far exceed the average (dashed line). (d) Visual heterogeneity under the shared semantic label "fish" in ImageNet-R hendrycks2021many. While clownfish and goldfish are visually similar, starfish differs substantially in appearance despite sharing the same semantic root.
• Figure 2: Overview of the CPL-NC framework. Given an input image $X$, CPL-NC extracts visual features $f_v$ and retrieves visual prototypes $v_c$ from a class-aware cache $\mathcal{C}$. Textual prototypes $t_c$ are refined to $\hat{t}_c$ during test time. The cache dynamically adjusts class capacity $M_c$ based on frequency $p_c$ and inactivity, applying a decay-aware boost $M_c^{\text{boost}}$ for rare classes. Final predictions combine $f_v$, $v_c$, and $\hat{t}_c$, with cache updates triggered by low entropy. A negative contrastive module selects hard negatives $(v_c^{-}, t_c^{-})$ to enforce InfoNCE loss $\mathcal{L}_{\text{NCL}}$, improving semantic separation by updating only textual prototypes.
• Figure 3: (a) Impact of update frequency on model performance evaluated on Aircraft, and (b) ImageNet; (c) influence of the class-aware factor $\gamma$ on ImageNet, ImageNet-A, and Aircraft; (d) effect of the NCL loss weight on model generalization across ImageNet-R, DTD, and Pets datasets.