RevCD -- Reversed Conditional Diffusion for Generalized Zero-Shot Learning

TL;DR

This work tackles generalized zero-shot learning by introducing RevCD, a reversed conditional diffusion model that generates semantic embeddings conditioned on visual inputs to bridge seen and unseen classes. By modeling the semantic posterior with a diffusion process conditioned on $x$, and optimizing a composite objective $\\mathcal{L}_{total} = \lambda_1 \\mathcal{L}_{rec} + \lambda_2 \\mathcal{L}_{noise} + \lambda_3 \\mathcal{L}_{classification}$, RevCD achieves strong seen-class accuracy while maintaining competitive unseen-class performance across AwA2, CUB, and SUN datasets. The architecture uses cross Hadamard-Addition embeddings, time-dependent sinusoidal encodings, and a visual-conditioned U-Net, enabling classifier-free guidance during sampling. Experimental results demonstrate the promise of diffusion-based density estimation as a back-end for GZSL, with notable gains on semantically diverse datasets and robust cross-dataset performance. The work provides open-source code and lays groundwork for diffusion-centric approaches in zero-shot settings.

Abstract

In Generalized Zero-Shot Learning (GZSL), we aim to recognize both seen and unseen categories using a model trained only on seen categories. In computer vision, this translates into a classification problem, where knowledge from seen categories is transferred to unseen categories by exploiting the relationships between visual features and available semantic information, such as text corpora or manual annotations. However, learning this joint distribution is costly and requires one-to-one training with corresponding semantic information. We present a reversed conditional Diffusion-based model (RevCD) that mitigates this issue by generating semantic features synthesized from visual inputs by leveraging Diffusion models' conditional mechanisms. Our RevCD model consists of a cross Hadamard-Addition embedding of a sinusoidal time schedule and a multi-headed visual transformer for attention-guided embeddings. The proposed approach introduces three key innovations. First, we reverse the process of generating semantic space based on visual data, introducing a novel loss function that facilitates more efficient knowledge transfer. Second, we apply Diffusion models to zero-shot learning - a novel approach that exploits their strengths in capturing data complexity. Third, we demonstrate our model's performance through a comprehensive cross-dataset evaluation. The complete code will be available on GitHub.

Figures (4)

• Figure 1: Overview of the proposed RevCD model: We train a denoising process using only seen samples (indicated by green boxes). Once trained, the model can estimate the semantic distribution by conditioning on the visual space of unseen samples (represented by blue boxes) and Gaussian noise. The final classification is conducted through a simple nearest-neighbor search based on the estimated density.
• Figure 2: The figure illustrates our proposed approach for training. (a) presents our high-level architecture and associated loss functions. By conditioning the image, we can infer the semantic distribution of unseen classes. (b) provides a detailed view of our U-net architecture. It implements sinusoidal time and cross-Hadamard-Addition conditional embeddings for optimal control over the learned distribution. In ZSL, the goal is to transfer the knowledge of how to infer the distribution rather than the distribution itself.
• Figure 3: The cosine distance to the true semantic space and the denoised learned representation are shown for both the AWA dataset and the CUB dataset. (a) For AWA, we observed a rapid reduction in noise in the initial timesteps, which gradually slowed as it approached the fully denoised space. (b) Conversely, for the CUB dataset, which possesses a semantically fine-grained space, the reduction in noise exhibited an inverse pattern.
• Figure 4: Seen and unseen accuracy as a function of the classification loss weight $\lambda_3$ across AWA, CUB, and SUN. Increasing $\lambda_3$ consistently improves seen class accuracy but degrades unseen performance, with the trade-off varying by dataset. AWA exhibits sharp overfitting due to coarse semantics, while CUB remains more stable under stronger supervision. SUN shows a more balanced response, reflecting its broader semantic diversity.