Follow-Up Differential Descriptions: Language Models Resolve Ambiguities for Image Classification

TL;DR

This paper addresses zero-shot image classification with vision-language models by tackling inter-class ambiguities through dataset-tailored natural language descriptions. The authors propose FuDD, which first detects ambiguous targets from initial predictions, then generates differential descriptions for those ambiguities using a Large Language Model, and finally performs a follow-up classification using these descriptions. Empirical results across 12 datasets show FuDD consistently improves over generic description ensembles and naive LLM prompts, with gains up to 13.95 percentage points and competitive performance to few-shot methods when high-quality descriptions are produced. The work demonstrates the potential of leveraging LLMs to tailor class representations to downstream datasets, reducing ambiguity-driven errors without labeled data.

Abstract

A promising approach for improving the performance of vision-language models like CLIP for image classification is to extend the class descriptions (i.e., prompts) with related attributes, e.g., using brown sparrow instead of sparrow. However, current zero-shot methods select a subset of attributes regardless of commonalities between the target classes, potentially providing no useful information that would have helped to distinguish between them. For instance, they may use color instead of bill shape to distinguish between sparrows and wrens, which are both brown. We propose Follow-up Differential Descriptions (FuDD), a zero-shot approach that tailors the class descriptions to each dataset and leads to additional attributes that better differentiate the target classes. FuDD first identifies the ambiguous classes for each image, and then uses a Large Language Model (LLM) to generate new class descriptions that differentiate between them. The new class descriptions resolve the initial ambiguity and help predict the correct label. In our experiments, FuDD consistently outperforms generic description ensembles and naive LLM-generated descriptions on 12 datasets. We show that differential descriptions are an effective tool to resolve class ambiguities, which otherwise significantly degrade the performance. We also show that high quality natural language class descriptions produced by FuDD result in comparable performance to few-shot adaptation methods.

Figures (7)

• Figure 1: A) Attributes for three different classes. B) Two sample classification tasks involving the wren class. The distinguishing characteristics of each class vary based on other classes. Our approach selects the class descriptions based on other classes in the dataset to provide the information that differentiates the target classes.
• Figure 2: FuDD overview. A) Using the model's initial prediction, we identify the potentially ambiguous classes. B) We use a large language model to generate class descriptions that differentiate the ambiguous classes. C) We use the new differential descriptions in a follow-up classification task to resolve the initial ambiguity and select the correct label.
• Figure 3: Impact of differential descriptions for $k$ most ambiguous classes with ViT-L/14@336px. $k{=}1$ is accuracy with a single template. Providing differentiating details for the most ambiguous classes accounts for most of FuDD's gains, with diminishing gains for less ambiguous classes.
• Figure 4: Top-5 most common attributes described by Llama 2 before and after fine-tuning. The fine-tuned model describes a more diverse and visually differentiating set of attributes.
• Figure 5: Descriptions generated by Llama 2 before and after fine-tuning.