GET: Unlocking the Multi-modal Potential of CLIP for Generalized Category Discovery

TL;DR

GET tackles generalized category discovery by unlocking CLIP's multi-modal potential through a Text Embedding Synthesizer (TES) that generates pseudo text embeddings for unlabelled data and a dual-branch framework with cross-modal instance consistency. TES aligns pseudo text with visual features and distills toward real text embeddings, enabling seamless use of CLIP's text encoder without external corpora. The two-branch system mutualizes visual and semantic information via cross-modal knowledge exchange, yielding state-of-the-art results across fine-grained and cross-domain GCD benchmarks and demonstrating robust performance when CLIP encounters unseen categories. Overall, GET advances practical multi-modal GCD by integrating text-based discriminative signals into category discovery, improving discrimination of visually similar classes and broadening CLIP's applicability to open-world settings.

Abstract

Given unlabelled datasets containing both old and new categories, generalized category discovery (GCD) aims to accurately discover new classes while correctly classifying old classes. Current GCD methods only use a single visual modality of information, resulting in a poor classification of visually similar classes. As a different modality, text information can provide complementary discriminative information, which motivates us to introduce it into the GCD task. However, the lack of class names for unlabelled data makes it impractical to utilize text information. To tackle this challenging problem, in this paper, we propose a Text Embedding Synthesizer (TES) to generate pseudo text embeddings for unlabelled samples. Specifically, our TES leverages the property that CLIP can generate aligned vision-language features, converting visual embeddings into tokens of the CLIP's text encoder to generate pseudo text embeddings. Besides, we employ a dual-branch framework, through the joint learning and instance consistency of different modality branches, visual and semantic information mutually enhance each other, promoting the interaction and fusion of visual and text knowledge. Our method unlocks the multi-modal potentials of CLIP and outperforms the baseline methods by a large margin on all GCD benchmarks, achieving new state-of-the-art. Our code is available at: https://github.com/enguangW/GET.

Figures (12)

• Figure 1: The motivation of our method. (a) Current GCD methods wen2023parametric rely on single visual features, resulting in poor classification of visually similar classes; Our approach introduces text information, improving the discriminative capabilities of the model. (b) Our proposed method maps image embeddings to text embeddings while simultaneously achieving modal alignment.
• Figure 2: Overview of our GET framework. (a) In the first stage, we introduce a text embedding synthesizer that generates pseudo text embeddings for unlabelled data. TES learns a linear mapping that transforms image features into input tokens for the text encoder. The resulting pseudo text embeddings are then used for joint training in the second stage. (b) We proposed a dual-branch multi-modal joint training framework with a cross-modal instance consistency objective in the second stage. Two branches utilize the same parameterized training strategy wen2023parametric while focusing on text and visual information, respectively. (c) Our cross-modal instance consistency objective makes visual and text information exchange and benefit from each other.
• Figure 3: Attention map visualization of class tokens.
• Figure 4: The t-SNE visualizations.
• Figure 5: Experiments on the pseudo-tokens and layers in TES.