Dual Prototype Evolving for Test-Time Generalization of Vision-Language Models

TL;DR

This work introduces Dual Prototype Evolving (DPE), a novel test-time adaptation approach for VLMs that effectively accumulates task-specific knowledge from multi-modalities and consistently outperforms previous state-of-the-art methods while also exhibiting competitive computational efficiency.

Abstract

Test-time adaptation, which enables models to generalize to diverse data with unlabeled test samples, holds significant value in real-world scenarios. Recently, researchers have applied this setting to advanced pre-trained vision-language models (VLMs), developing approaches such as test-time prompt tuning to further extend their practical applicability. However, these methods typically focus solely on adapting VLMs from a single modality and fail to accumulate task-specific knowledge as more samples are processed. To address this, we introduce Dual Prototype Evolving (DPE), a novel test-time adaptation approach for VLMs that effectively accumulates task-specific knowledge from multi-modalities. Specifically, we create and evolve two sets of prototypes--textual and visual--to progressively capture more accurate multi-modal representations for target classes during test time. Moreover, to promote consistent multi-modal representations, we introduce and optimize learnable residuals for each test sample to align the prototypes from both modalities. Extensive experimental results on 15 benchmark datasets demonstrate that our proposed DPE consistently outperforms previous state-of-the-art methods while also exhibiting competitive computational efficiency. Code is available at https://github.com/zhangce01/DPE-CLIP.

Figures (6)

• Figure 1: Comparison of our DPE with zero-shot CLIP radford2021learning, TPT shu2022test, and TDA karmanov2024efficient. We denote CLIP's parallel textual and visual encoders as $\mathcal{E}_t$ and $\mathcal{E}_v$, respectively. While previous methods solely adapt the CLIP model from a single modality, we design our DPE to evolve prototypes from both textual and visual modalities to progressively capture more accurate multi-modal representations for target classes during test time.
• Figure 2: An overview of our DPE method. We introduce prototypes from both textual and visual modalities and enable prototype-based inference with CLIP. For each test sample, we optimize both prototypes using learnable residual parameters with alignment loss $\mathcal{L}_{\mathsf{align}}$ and self-entropy loss $\mathcal{L}_{\mathsf{aug}}$. These prototypes are also progressively evolved over time to capture more accurate and discriminative multi-modal representations for target classes.
• Figure 3: t-SNE van2008visualizing visualizations of the stored image features in the priority queues. With more samples getting in, the selected image features from each class become more clustered, leading to more representative visual prototypes.
• Figure 4: Ablation studies. (Left) Sensitivity analysis of $\tau_t$ and $M$ on Caltech101 fei2004learning; (Middle) Analysis of the performance contributions from various learnable parameter settings across three datasets; (Right) Performance on three datasets with varying scale factor $\lambda$ in Equation (\ref{['eq:update']}).
• Figure A1: Effectiveness of different algorithm components. VPE, TPE, and PRL refer to visual prototype evolution, textual prototype evolution, and prototype residual learning, respectively.