On the test-time zero-shot generalization of vision-language models: Do we really need prompt learning?

TL;DR

This paper tackles zero-shot generalization in vision-language models by proposing a training-free test-time augmentation method, MTA, based on a robust MeanShift formulation. MTA jointly optimizes an inlierness score and a density mode over multiple augmented views of an image, using a text-informed affinity and an entropy barrier to downweight outliers, all without updating prompts or model parameters. Across 15 datasets, MTA outperforms test-time prompt tuning baselines while delivering substantially faster runtimes and strong compatibility with both zero-shot and few-shot setups. The approach provides a practical, black-box-friendly alternative to prompt learning for deploying vision-language models in real-world API and edge scenarios.

Abstract

The development of large vision-language models, notably CLIP, has catalyzed research into effective adaptation techniques, with a particular focus on soft prompt tuning. Conjointly, test-time augmentation, which utilizes multiple augmented views of a single image to enhance zero-shot generalization, is emerging as a significant area of interest. This has predominantly directed research efforts toward test-time prompt tuning. In contrast, we introduce a robust MeanShift for Test-time Augmentation (MTA), which surpasses prompt-based methods without requiring this intensive training procedure. This positions MTA as an ideal solution for both standalone and API-based applications. Additionally, our method does not rely on ad hoc rules (e.g., confidence threshold) used in some previous test-time augmentation techniques to filter the augmented views. Instead, MTA incorporates a quality assessment variable for each view directly into its optimization process, termed as the inlierness score. This score is jointly optimized with a density mode seeking process, leading to an efficient training- and hyperparameter-free approach. We extensively benchmark our method on 15 datasets and demonstrate MTA's superiority and computational efficiency. Deployed easily as plug-and-play module on top of zero-shot models and state-of-the-art few-shot methods, MTA shows systematic and consistent improvements.

Figures (5)

• Figure 1: Interpretation of Eqs. \ref{['final-updates-y']} and \ref{['fixed-point-iteration-updates']} in (a) and (b) respectively. Our robust MeanShift alternatively solves these 2 equations until convergence. A pseudocode is available in Appendix \ref{['appendix:further_details']}.
• Figure 2: Runtime in seconds per image on ImageNet for TPT and MTA with 5 different backbones: RN50 (ResNet-50), RN101 (ResNet-101), ViT-B/32, ViT-B/16 and ViT-L/14. Experiences were performed on a single A100 40Gb GPU.
• Figure 3: Few-shot learning results on the 10 fine-grained datasets and ImageNet. We compare MTA and TPT when added on top of CoOp prompts (M=4 tokens) for increasing number of shots. Averaged top-1 accuracy over the 11 datasets is shown on the bottom right, we additionally show the averaged top-1 accuracy for CoOp with M=16 tokens.
• Figure 4: Averaged top-1 accuracy of MTA with and without majority vote for final prediction on the 5 ImageNet variants with increasing number of augmented views.
• Figure 5: Additional results for Figure \ref{['fig:coop_few_shots']} with M=16 tokens for the CoOp pretrained prompts.