Embracing Diversity: Interpretable Zero-shot classification beyond one vector per class

TL;DR

This work identifies a fundamental limitation of one-vector-per-class zero-shot classification in vision-language models when objects exhibit substantial intra-class diversity. It introduces a two-step approach that first infers diverse class attributes with a large language model and then consolidates image-to-subpopulation similarities nonlinearly by attending to the top-$k$ related subpopulations, yielding per-class scores. Across a broad suite of datasets and two VLMs, the method matches or surpasses strong baselines and achieves notable improvements on hardest classes and real-world fairness metrics, while also offering faithful, interpretable explanations for each decision. The proposed framework is scalable to many attributes and provides tunable trade-offs between overall and worst-case accuracy, supporting transparent and robust zero-shot classification in diverse real-world settings.

Abstract

Vision-language models enable open-world classification of objects without the need for any retraining. While this zero-shot paradigm marks a significant advance, even today's best models exhibit skewed performance when objects are dissimilar from their typical depiction. Real world objects such as pears appear in a variety of forms -- from diced to whole, on a table or in a bowl -- yet standard VLM classifiers map all instances of a class to a \it{single vector based on the class label}. We argue that to represent this rich diversity within a class, zero-shot classification should move beyond a single vector. We propose a method to encode and account for diversity within a class using inferred attributes, still in the zero-shot setting without retraining. We find our method consistently outperforms standard zero-shot classification over a large suite of datasets encompassing hierarchies, diverse object states, and real-world geographic diversity, as well finer-grained datasets where intra-class diversity may be less prevalent. Importantly, our method is inherently interpretable, offering faithful explanations for each inference to facilitate model debugging and enhance transparency. We also find our method scales efficiently to a large number of attributes to account for diversity -- leading to more accurate predictions for atypical instances. Finally, we characterize a principled trade-off between overall and worst class accuracy, which can be tuned via a hyperparameter of our method. We hope this work spurs further research into the promise of zero-shot classification beyond a single class vector for capturing diversity in the world, and building transparent AI systems without compromising performance.

Figures (13)

• Figure 1: We test models on datasets that provide groundtruth attributes (shown in bold) annotating hierarchies, diverse states, and real-world shifts (e.g., rojas2022the labels the income level and country of origin of each image, towards promoting AI models that reduce bias) within a class. We find that standard zero-shot accuracy ('Base Acc.' above) drops significantly when certain attributes are present, namely when the attribute manifests in visual differences from what the model considers 'typical' for the class. We design our method to improve performance on these 'atypical' instances.
• Figure 2: The average precision (AP) of a classname embedding is often much lower than the average precision of a subpopulation (i.e. classname with attribute) embedding. Subpopulations that see large increases in AP by including the attribute tend to be atypical. We design our method to improve accuracy on these diverse subpopulations, by inferring and explicitly accounting for them.
• Figure 3: An Arcticfox can more closely resemble a typical wolf than a typical fox. Standard zero-shot classification using one vector per class (the classname embedding) is ill suited for this case. We address this issue by nonlinearly consolidating similarities to multiple vectors per class that explicitly encode the diverse subpopulations within the class. See section \ref{['sec:consolidation']} for full explanation.
• Figure 4: Instances where our method corrects mistakes of the standard approach. The attributes used in inference also serve as faithful fine-grained explanations. Notably, these samples are atypical, suggesting that inspecting samples where our method and standard classification disagree can enable automatic surfacing of atypical cases, towards better understanding the task at hand.
• Figure 5: Accuracy, overall and for the worst classes, as new types of attributes are added. Performance for our consolidation scheme continuously improves, while it saturates or deteriorates for others. Figure \ref{['app-fig:add_in_attr']} shows similar trends for accuracy on the worst 20% of classes and subpopulations.