Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models

TL;DR

This paper tackles zero-shot generalization in vision–language models by introducing Test-Time Prompt Tuning (TPT), which adapts prompts on a per-test-sample basis without requiring task-specific training data. TPT learns prompts through an entropy-based objective over multiple augmented views and employs a confidence filter to discard noisy views, enabling robust zero-shot performance on image classification and context-dependent reasoning tasks like Bongard-HOI. Empirical results show TPT improves zero-shot CLIP performance on natural distribution shifts (average ~3.6% gain; up to 6.9% on ImageNet-A) and matches or exceeds state-of-the-art few-shot prompt tuning on cross-dataset generalization, while also achieving strong results in context-dependent visual reasoning. Overall, TPT demonstrates that prompting strategies at test time can substantially enhance the generalization capabilities of vision-language foundation models without access to additional training data, with a clear path for broader application and efficiency considerations.

Abstract

Pre-trained vision-language models (e.g., CLIP) have shown promising zero-shot generalization in many downstream tasks with properly designed text prompts. Instead of relying on hand-engineered prompts, recent works learn prompts using the training data from downstream tasks. While effective, training on domain-specific data reduces a model's generalization capability to unseen new domains. In this work, we propose test-time prompt tuning (TPT), a method that can learn adaptive prompts on the fly with a single test sample. For image classification, TPT optimizes the prompt by minimizing the entropy with confidence selection so that the model has consistent predictions across different augmented views of each test sample. In evaluating generalization to natural distribution shifts, TPT improves the zero-shot top-1 accuracy of CLIP by 3.6% on average, surpassing previous prompt tuning approaches that require additional task-specific training data. In evaluating cross-dataset generalization with unseen categories, TPT performs on par with the state-of-the-art approaches that use additional training data. Project page: https://azshue.github.io/TPT.

Figures (9)

• Figure 1: Test-time Prompt Tuning (TPT) for image classification. We tune adaptive prompts on the fly with a single test sample, without the need for additional training data or annotations. TPT optimizes the prompt to encourage consistent predictions across augmented views by minimizing the marginal entropy. We introduce confidence selection to filter out noisy augmentations.
• Figure 2: Test-time Prompt Tuning (TPT) for context-dependent visual reasoning on Bongard-HOI benchmark. A test sample in Bonagrd-HOI consists of several support images that exemplify a visual concept, and the model needs to predict whether the query image contains the concept. TPT tunes the prompt and class tokens simultaneously on the support images using the cross-entropy loss.
• Figure 3: Cross-dataset improvement normalized by the zero-shot baseline performance. In each matrix $A$, $A_{i, j}$ is the normalized relative improvement on the $j_{th}$ dataset of using the prompt tuned on the $i$-th dataset. The value $A_{i, j}$ stands for how well a method trained on a source dataset $i$ performs on a target dataset $j$, in comparison with a zero-shot CLIP baseline (using a hand-crafted prompt). Thus, the higher, the better. The last row is the performance of TPT, which is not tuned on any source dataset. The last column summarizes the average improvement over 10 datasets, measuring the overall generalization ability across the 10 datasets.
• Figure 4: Ablating the effects of different components of TPT. We evaluate the top-1 accuracy on the distribution shifts benchmarks in section \ref{['sec:exp-ood']}. Methods are implemented based on a CLIP-RN50.
• Figure 5: Analysis on the trade-off between efficiency and accuracy. We evaluate the top-1 accuracy on the distribution shifts benchmarks in section \ref{['sec:exp-ood']}. Results are based on a CLIP-RN50.