Interpretable Zero-shot Learning with Infinite Class Concepts

TL;DR

This paper redefines class semantics in ZSL with a focus on transferability and discriminability, introducing a novel framework called Zero-shot Learning with Infinite Class Concepts (InfZSL), which leverages the powerful capabilities of LLMs to dynamically generate an unlimited array of phrase-level class concepts.

Abstract

Zero-shot learning (ZSL) aims to recognize unseen classes by aligning images with intermediate class semantics, like human-annotated concepts or class definitions. An emerging alternative leverages Large-scale Language Models (LLMs) to automatically generate class documents. However, these methods often face challenges with transparency in the classification process and may suffer from the notorious hallucination problem in LLMs, resulting in non-visual class semantics. This paper redefines class semantics in ZSL with a focus on transferability and discriminability, introducing a novel framework called Zero-shot Learning with Infinite Class Concepts (InfZSL). Our approach leverages the powerful capabilities of LLMs to dynamically generate an unlimited array of phrase-level class concepts. To address the hallucination challenge, we introduce an entropy-based scoring process that incorporates a ``goodness" concept selection mechanism, ensuring that only the most transferable and discriminative concepts are selected. Our InfZSL framework not only demonstrates significant improvements on three popular benchmark datasets but also generates highly interpretable, image-grounded concepts. Code will be released upon acceptance.

Figures (12)

• Figure 1: Motivation Illustration. (a) LLM-based document ZSL methods encounter two main issues: non-visual semantics caused by LLM hallucination and a lack of transparency in visual-semantic alignment. (b) Our InfZSL addresses these challenges by introducing Concept Entropy Selection and Scoring (CESS), which selects and scores concepts with high transferability and discriminability. InfZSL also enables transparent visual-semantic alignment, enhancing interpretability in the ZSL decision-making process.
• Figure 2: An illustration of our InfZSL. It consists of three steps. (a) Concept generation: we use our designed promt to extract any number of class concepts. (b) Concept selection and scoring: LLMs might generate non-visual concepts; however, even among visual concepts, some may possess only transferability (e.g. 'furry' in the illustration.) or discriminative power (e.g. 'horn'). Only the concepts that have both high discriminability and transferability are we need in ZSL (e.g. 'long neck'). We leverage our proposed concept entropy to select and score them to get the class embedding $\mathbf{s}_{m}$ based on LLM-generated concepts. (3) Once construct our $\mathbf{s}_{m}$, we can immediately integrate it with human-annotated class embedding $\mathbf{s}_{h}$ into existing concept-based embedding or generative ZSL methods.
• Figure 3: The heatmap visualization of our LLM-concept-based semantic embeddings.
• Figure 4: Hyper-parameters sensitivity analysis on (a) AWA2 and (b) CUB. The shaded area indicates the performance improvement compared to baseline.
• Figure 5: Attention visualizations of our LLM-generated concepts for two unseen classes.