Will It Zero-Shot?: Predicting Zero-Shot Classification Performance For Arbitrary Queries

TL;DR

The paper tackles predicting zero-shot classification performance of vision-language models like CLIP for arbitrary natural-language queries. It introduces an image-augmented, consistency-based evaluation framework that uses generated task-relevant imagery alongside text prompts to estimate zero-shot accuracy without labeled data, encapsulated in a compound score that combines consistency and silhouette components. The approach is validated across multiple CLIP benchmark datasets, showing that image-based scores correlate more strongly with true zero-shot accuracy than text-only proxies, and is operationalized through an interactive web tool for end users. Limitations are discussed via outliers (e.g., ObjectNet) and ambiguous class meanings, with future work focused on richer contextual prompts and user-driven refinement.

Abstract

Vision-Language Models like CLIP create aligned embedding spaces for text and images, making it possible for anyone to build a visual classifier by simply naming the classes they want to distinguish. However, a model that works well in one domain may fail in another, and non-expert users have no straightforward way to assess whether their chosen VLM will work on their problem. We build on prior work using text-only comparisons to evaluate how well a model works for a given natural language task, and explore approaches that also generate synthetic images relevant to that task to evaluate and refine the prediction of zero-shot accuracy. We show that generated imagery to the baseline text-only scores substantially improves the quality of these predictions. Additionally, it gives a user feedback on the kinds of images that were used to make the assessment. Experiments on standard CLIP benchmark datasets demonstrate that the image-based approach helps users predict, without any labeled examples, whether a VLM will be effective for their application.

Figures (5)

• Figure 1: Vision-Language Models (VLMs) enable users without machine learning expertise to rapidly deploy visual classifiers. These users need practical tools to determine if a VLM will be effective for their tasks. This paper introduces a simple approach for predicting zero-shot classification performance by analyzing the internal consistency of CLIP's text and image embedding spaces.
• Figure 2: Each scatter plot shows the relationship between zero-shot classification accuracy on the real dataset and our compound score in real (blue) or generated (orange) data. Each dot represents a class in datasets.
• Figure 3: Example images from classes where the compound scores do not match the CLIP Zero-Shot Accuracy. These outliers can be caused by real dataset images being weaker depictions of the class than the generated images (a) or image generation misunderstanding the class (b).
• Figure 4: Our tool predicts that CLIP would be able to correctly classify images of spotted lanternflies relative to images of similar-looking insects about 63% predicted accuracy. However, when the alternative classes are endangered and other threatened species in the northeast United States, CLIP is predicted to correctly classify a spotted lanternfly relative to the other insects about with 78% predicted accuracy.
• Figure 5: A comparison of images generated from diverse text captions (top) and images generated with the simple prompt of the class name "woman" (bottom). The text captions lead to more detailed and diverse images of the class which matters much more in general classes such as "woman".