Prompt-Driven Feature Diffusion for Open-World Semi-Supervised Learning

TL;DR

PDFD tackles Open-World Semi-Supervised Learning by integrating a feature-level diffusion model guided by class prototypes as prompts, enabling discriminative feature learning for seen and unseen classes. It combines a diffusion loss $\mathcal{L}_{\text{diff}}$ with a class-conditional adversarial loss $\mathcal{L}_{\text{adv}}$ and a distribution-aware pseudo-labeling strategy to leverage both labeled and unlabeled data. Class prototypes are computed from labeled data for seen classes and confidently predicted unlabeled data for unseen classes, and are used as prompts in a transformer-based diffusion model to generate class-specific features. Empirical results on CIFAR-10, CIFAR-100, and ImageNet-100 show PDFD achieving state-of-the-art performance across SSL, Open-Set SSL, Novel Class Discovery, and OW-SSL, demonstrating strong unseen-class recognition and robust knowledge transfer to novel categories.

Abstract

In this paper, we present a novel approach termed Prompt-Driven Feature Diffusion (PDFD) within a semi-supervised learning framework for Open World Semi-Supervised Learning (OW-SSL). At its core, PDFD deploys an efficient feature-level diffusion model with the guidance of class-specific prompts to support discriminative feature representation learning and feature generation, tackling the challenge of the non-availability of labeled data for unseen classes in OW-SSL. In particular, PDFD utilizes class prototypes as prompts in the diffusion model, leveraging their class-discriminative and semantic generalization ability to condition and guide the diffusion process across all the seen and unseen classes. Furthermore, PDFD incorporates a class-conditional adversarial loss for diffusion model training, ensuring that the features generated via the diffusion process can be discriminatively aligned with the class-conditional features of the real data. Additionally, the class prototypes of the unseen classes are computed using only unlabeled instances with confident predictions within a semi-supervised learning framework. We conduct extensive experiments to evaluate the proposed PDFD. The empirical results show PDFD exhibits remarkable performance enhancements over many state-of-the-art existing methods.

Figures (2)

• Figure 1: The proposed PDFD framework trained on $\mathcal{D}_l$, $\mathcal{D}_u$. The feature encoder $f$ takes as input the labeled data and unlabeled data to generate their learned embeddings. The embeddings of the labeled and unlabeled samples are used to calculate the class prototypes which are used as prompts for the diffusion model. The diffusion model, guided by the loss $\mathcal{L}_{\text{diff}}$, predicts the noise $\xi_\phi$ from noisy features. Concurrently, the classifier $h$ and encoder $f$ are trained, aiming to minimize the supervised loss $\mathcal{L}_{\text{ce}}^{l}$ and the pseudo-labeling loss $\mathcal{L}_{\text{ce}}^{u}$. Additionally, a class conditional-adversarial training component is integrated, wherein the generator $\xi_{\phi}$ aims to produce feature representations that successfully mislead the discriminator $D_{\psi}$, assessed by the adversarial loss $\mathcal{L}_{\text{adv}}$, into categorizing them as real features.
• Figure 2: Pseudo-Label Selection Analysis. (a) Confidence difference between seen and unseen classes during the training on CIFAR-100 (b) Effect of distribution-aware pseudo-label selection on learning unseen classes during the training on CIFAR-100.