Unsupervised Prompt Learning for Vision-Language Models

TL;DR

This work tackles prompt design in vision-language models by removing the need for labeled target data. It introduces unsupervised prompt learning (UPL), which first generates pseudo labels on unlabeled data using a simple CLIP prompt and then learns a shared continuous prompt representation via self-training, with optional pseudo-label and prompt ensembles to boost quality. Across 11 datasets, UPL improves over the original CLIP with handcrafted prompts, and UPL*—which aggregates multiple CLIP models for pseudo-labeling—achieves further gains, rivaling 8-shot prompting methods. The approach reduces the reliance on labeled data for prompt engineering while achieving strong transfer performance, and it generalizes to other prompt-learning frameworks.

Abstract

Contrastive vision-language models like CLIP have shown great progress in transfer learning. In the inference stage, the proper text description, also known as prompt, needs to be carefully designed to correctly classify the given images. In order to avoid laborious prompt engineering, recent works such as CoOp, CLIP-Adapter and Tip-Adapter propose to adapt vision-language models for downstream image recognition tasks on a small set of labeled data. Though promising improvements are achieved, requiring labeled data from the target datasets may restrict the scalability. In this paper, we explore a different scenario, in which the labels of the target datasets are unprovided, and we present an unsupervised prompt learning (UPL) approach to avoid prompt engineering while simultaneously improving transfer performance of CLIP-like vision-language models. As far as we know, UPL is the first work to introduce unsupervised learning into prompt learning. Experimentally, our UPL outperforms original CLIP with prompt engineering on ImageNet as well as other 10 datasets. An enhanced version of UPL is even competitive with the 8-shot CoOp and the 8-shot TIP-Adapter on most datasets. Code and models are available at https://github.com/tonyhuang2022/UPL.

Figures (9)

• Figure 1: (a) Inference of the pre-trained CLIP. (b) Existing methods such as CoOp, CLIP-Adapter and Tip-Adapter use a small set of labeled images from target datasets to adapt pre-trained CLIP to downstream task. (c) Our UPL conducts prompt learning on unlabeled images from target datasets.
• Figure 2: Overview of the proposed unsupervised prompt learning (UPL) framework. Our UPL mainly contains two parts, namely pseudo label generation and prompt representation optimization. We first use CLIP with a simple prompt (e.g., "a photo of a [CLS]") to generate pseudo labels for target datasets and select top-$K$ confident samples per class for subsequent training. Then we define a learnable prompt representation which is optimized on selected pseudo-labeled samples. For inference, we simply swap out the hand-crafted prompts with the well-optimized prompt representations.
• Figure 3: Analysis of two pseudo-labeling strategies on UCF-101 dataset. We observe CLIP shows biased preferences for different classes in transfer learning. Classical self-training pre-defines a threshold to select samples of high probability, resulting in an imbalanced distribution of pseudo-labeled data (orange and green lines). We advocate to select top-$K$ confident samples per class to generate a balanced set of pseudo-labeled data for self-training (blue line).
• Figure 4: We select top-$16$ confident samples per class on UCF-101 dataset and compute averaged probability and pseudo label accuracy for each class. We observe that probability (confidence) can not completely reflect the quality of pseudo labels. It is possible for categories with low averaged probabilities to have accurate pseudo labels.
• Figure 5: CLIP models with different vision encoders have preferences for different classes. We study this phenomenon on UCF-101 dataset. We compare three CLIP models, namely ResNet-50, ResNet-50x64 and ViT-L/14 and compute the class-wise pseudo label accuracy for each model. We show the accuracy gap between ResNet-50x64 and ResNet-50 (blue line), and the accuracy gap between ViT-L/14 and ResNet-50 (orange line).