Discriminative Image Generation with Diffusion Models for Zero-Shot Learning

TL;DR

The paper tackles the interpretability and scalability limitations of traditional generative zero-shot learning by introducing DIG-ZSL, which learns discriminative class tokens (DCT) for unseen classes under a category discrimination model (CDM) and uses text prompts in a Stable Diffusion pipeline to generate highly discriminative unseen-class images. The approach integrates a CDM trained on seen data with CLIP-based semantic prototypes, iteratively optimizes per-unseen-class tokens, and trains a ZSL classifier on a combination of real and synthetic data, with an early-stopping strategy and bias-calibrated inference. Empirical results on four benchmarks (AWA2, CUB, FLO, SUN) show that DIG-ZSL substantially surpasses non-human-annotated prototype methods (average Top-1 improvement of $24.2\%$) and is competitive with methods using human-annotated prototypes, while providing improved image quality (lower $FID$) and clearer discriminative signals. Overall, the work demonstrates a practical and scalable path to interpretable, diffusion-based discriminative image generation for zero-shot learning with potential impact on broader diffusion-assisted recognition tasks.$

Abstract

Generative Zero-Shot Learning (ZSL) methods synthesize class-related features based on predefined class semantic prototypes, showcasing superior performance. However, this feature generation paradigm falls short of providing interpretable insights. In addition, existing approaches rely on semantic prototypes annotated by human experts, which exhibit a significant limitation in their scalability to generalized scenes. To overcome these deficiencies, a natural solution is to generate images for unseen classes using text prompts. To this end, We present DIG-ZSL, a novel Discriminative Image Generation framework for Zero-Shot Learning. Specifically, to ensure the generation of discriminative images for training an effective ZSL classifier, we learn a discriminative class token (DCT) for each unseen class under the guidance of a pre-trained category discrimination model (CDM). Harnessing DCTs, we can generate diverse and high-quality images, which serve as informative unseen samples for ZSL tasks. In this paper, the extensive experiments and visualizations on four datasets show that our DIG-ZSL: (1) generates diverse and high-quality images, (2) outperforms previous state-of-the-art nonhuman-annotated semantic prototype-based methods by a large margin, and (3) achieves comparable or better performance than baselines that leverage human-annotated semantic prototypes. The codes will be made available upon acceptance of the paper.

Figures (11)

• Figure 1: Schematic of the ZSL method. (a) Previous works (e.g., f-VAEGAN xian2019f) learn a generative model to synthesize visual features for unseen classes, conditioned on human-annotated semantic information and features from seen classes. These approaches results in a lack of interpretable insights. (b) Our proposed DIG-ZSL learns discriminative class tokens for unseen classes to generate images for unseen classes, based on a text-to-image diffusion model (e.g., Stable Diffusion (SD) rombach2022high) and class names. Our method enables generating diverse and high-quality images that contain discriminative information.
• Figure 2: Illustration of our proposed DIG-ZSL framework. We first learn a category discrimination model (CDM) with the training data from seen classes and semantic prototypes derived from class names. Then, we initialize a token $S_*$, and iteratively modifying the embedding of this input token (denoted as $\boldsymbol{e}_*$) under the guidance of CDM, using a cross-entropy loss. Equipped with the optimized token for each unseen category, we incorporate it into the conditioning prompt to generate realistic and discriminative images. Finally, we combine these synthesized data with real training data to learn an effective ZSL classifier.
• Figure 3: t-SNE visualization of the DCT embeddings. A dot indicates a DCT embedding for an unseen category from CUB dataset. Red dot and green dot indicate the tokens that are clustered together. (Please zoom in for details.)
• Figure 4: Visualization of generated images at the identical step for both the plain Stable Diffusion (SD) and DIG-ZSL. Our DIG-ZSL demonstrates the capability to enhance text-to-image alignment and significantly improve the portrayal of details. We use CUB as an example, with the green box indicating correct details, while the red box represents incorrect details.
• Figure 5: Hyper-parameter sensitivity. Take CUB dataset as an example, We explore (a) the effect of the number of generated images $N_{gen}$, and (b) the effect of calibration coefficient $\lambda$.