GenZSL: Generative Zero-Shot Learning Via Inductive Variational Autoencoder

TL;DR

GenZSL tackles zero-shot learning by inducing unseen-class features from semantically similar seen classes rather than imagining from scratch. It introduces an inductive variational autoencoder (IVAE) guided by two criteria: class diversity promotion (CDP) to decorrelate class semantic vectors, and target class-guided information boosting through reconstruction and entropy-maximizing losses. A semantically similar sample selection module and CLIP-based weak vectors enable effective induction without expert attributes, yielding strong gains over imagination-based generative ZSL on CUB, SUN, and AWA2, including around 24.7% improvement on AWA2 and significant training-speedups. Overall, GenZSL bridges classic ZSL with vision-language models, providing a flexible, scalable approach for scene-oriented zero-shot recognition.

Abstract

Remarkable progress in zero-shot learning (ZSL) has been achieved using generative models. However, existing generative ZSL methods merely generate (imagine) the visual features from scratch guided by the strong class semantic vectors annotated by experts, resulting in suboptimal generative performance and limited scene generalization. To address these and advance ZSL, we propose an inductive variational autoencoder for generative zero-shot learning, dubbed GenZSL. Mimicking human-level concept learning, GenZSL operates by inducting new class samples from similar seen classes using weak class semantic vectors derived from target class names (i.e., CLIP text embedding). To ensure the generation of informative samples for training an effective ZSL classifier, our GenZSL incorporates two key strategies. Firstly, it employs class diversity promotion to enhance the diversity of class semantic vectors. Secondly, it utilizes target class-guided information boosting criteria to optimize the model. Extensive experiments conducted on three popular benchmark datasets showcase the superiority and potential of our GenZSL with significant efficacy and efficiency over f-VAEGAN, e.g., 24.7% performance gains and more than $60\times$ faster training speed on AWA2. Codes are available at https://github.com/shiming-chen/GenZSL.

Figures (10)

• Figure 1: Motivation illustration. (a) Existing generative ZSL methods merely generate (imagine) the visual features from scratch guided by the strong class semantic vectors, resulting in suboptimal generative performance and scene generalization. For example, the generator inevitably generates similar classes of "Zebra" or others, e.g., "Donkey". (b) Our GenZSL generates (induces) the reliable visual features of unseen classes from the similar seen classes with the clues of weak class semantic vector, e.g., from "Horse" to "Zebra".
• Figure 2: Pipeline of our GenZSL. GenZSL first takes class diversity promotion to reduce the redundant information from class semantic vectors, and to improve the identity for all class semantic vectors. Then, it employs a semantically similar sample selection module to select the top-$k$ referent class from the seen classes for each target class as training inputs. Based on the referent samples, GenZSL learns an inductive variational autoencoder to create the new informative feature samples for unseen classes via induction optimized by target class-guided information boosting criteria.
• Figure 3: Class semantic vectors' similarity heatmaps are extracted by CLIP text encoder and CLIP with class diversity promotion on the CUB dataset. The similarity heatmaps on SUN and AWA2 are presented in Appendix \ref{['appdx-B']}.
• Figure 4: Qualitative evaluation with t-SNE visualization. The sample features from f-VAEGAN Xian2019FVAEGAND2AF are shown on the left, and from our GenZSL are shown on the right. We use 10 colors to denote randomly selected 10 classes from SUN. The "$\times$" and "$\circ$" are denoted as the real and synthesized sample features, respectively. The synthesized sample features and the real features distribute differently on the left while distributing similarly on the right. The visualization on the CUB and AWA2 is shown in Appendix \ref{['appdx-D']}.
• Figure 5: Hyper-parameter analysis. We show the performance variations on CUB by adjusting the value of loss weight $\lambda$ in (a), the number of the top referent classes top-$k$ in (b), and the number of synthesized samples of each unseen class $N_{syn}$ in (c).