GraphVL: Graph-Enhanced Semantic Modeling via Vision-Language Models for Generalized Class Discovery

TL;DR

GraphVL, a novel approach for vision-language modeling in GCD, leveraging CLIP is introduced, which integrates a graph convolutional network with CLIP’s text encoder to preserve class neighborhood structure and represents unlabeled samples based on their semantic distance to class prompts from the GCN, enabling semi-supervised clustering for class discovery and minimizing errors.

Abstract

Generalized Category Discovery (GCD) aims to cluster unlabeled images into known and novel categories using labeled images from known classes. To address the challenge of transferring features from known to unknown classes while mitigating model bias, we introduce GraphVL, a novel approach for vision-language modeling in GCD, leveraging CLIP. Our method integrates a graph convolutional network (GCN) with CLIP's text encoder to preserve class neighborhood structure. We also employ a lightweight visual projector for image data, ensuring discriminative features through margin-based contrastive losses for image-text mapping. This neighborhood preservation criterion effectively regulates the semantic space, making it less sensitive to known classes. Additionally, we learn textual prompts from known classes and align them to create a more contextually meaningful semantic feature space for the GCN layer using a contextual similarity loss. Finally, we represent unlabeled samples based on their semantic distance to class prompts from the GCN, enabling semi-supervised clustering for class discovery and minimizing errors. Our experiments on seven benchmark datasets consistently demonstrate the superiority of GraphVL when integrated with the CLIP backbone.

Figures (5)

• Figure 1: Architecture overview of GraphVL: We propose a novel approach that integrates a learnable GCN-based text projector on top of CLIP's text encoder $f_t$, alongside a visual projector $P_v$ built on top of CLIP's frozen vision encoder $f_v$. The model is trained using various metric losses, including one in the non-semantic space ($\mathcal{L}_{SDP}$), visual-semantic spaces ($\mathcal{L}_{CMA}$), and contextual similarity ($\mathcal{L}_{CS}$). During inference, we apply a semi-supervised clustering technique to cluster the unlabeled data, utilizing the similarity distributions relative to the learned class embeddings as clustering features ($\mathcal{Q}$).
• Figure 2: Comparing the number of trainable parameters in GraphVL and state-of-the-art class discovery methods.
• Figure 3: The sensitivity analysis of the loss terms and alpha using bar graph (left) and line graph (right), respectively.
• Figure 4: t-SNE plots clustering the classes in StanfordCars (left) and CIFAR-10 (right) using our proposed GraphVL.
• Figure 5: T-SNE plots between the image feature extracted from the GraphVL for different values of margin in the $\mathcal{L}_{CMA}$ loss.