SpectralGCD: Spectral Concept Selection and Cross-modal Representation Learning for Generalized Category Discovery

TL;DR

SpectralGCD is proposed, an efficient and effective multimodal approach to GCD that uses CLIP cross-modal image-concept similarities as a unified cross-modal representation and Spectral Filtering which exploits a cross-modal covariance matrix over the softmaxed similarities measured by a strong teacher model to automatically retain only relevant concepts from the dictionary.

Abstract

Generalized Category Discovery (GCD) aims to identify novel categories in unlabeled data while leveraging a small labeled subset of known classes. Training a parametric classifier solely on image features often leads to overfitting to old classes, and recent multimodal approaches improve performance by incorporating textual information. However, they treat modalities independently and incur high computational cost. We propose SpectralGCD, an efficient and effective multimodal approach to GCD that uses CLIP cross-modal image-concept similarities as a unified cross-modal representation. Each image is expressed as a mixture over semantic concepts from a large task-agnostic dictionary, which anchors learning to explicit semantics and reduces reliance on spurious visual cues. To maintain the semantic quality of representations learned by an efficient student, we introduce Spectral Filtering which exploits a cross-modal covariance matrix over the softmaxed similarities measured by a strong teacher model to automatically retain only relevant concepts from the dictionary. Forward and reverse knowledge distillation from the same teacher ensures that the cross-modal representations of the student remain both semantically sufficient and well-aligned. Across six benchmarks, SpectralGCD delivers accuracy comparable to or significantly superior to state-of-the-art methods at a fraction of the computational cost. The code is publicly available at: https://github.com/miccunifi/SpectralGCD.

Figures (9)

• Figure 1: Motivation and overview.(Left) Unimodal methods are efficient, but tend to overfit to spurious visual cues. (Center) Introducing textual supervision improves generalization, but increases computational overhead. (Right) SpectralGCD leverages an agnostic dictionary by selecting task-relevant concepts, improving generalization while remaining as efficient as unimodal techniques. While existing multimodal approaches treat modalities independently, our method uses a unified, cross-modal representation to train the classifier.
• Figure 2: The SpectralGCD two-phase approach.(a) Spectral Filtering uses the cross-modal covariance computed from teacher cross-modal representations to retain only its most informative components and isolate semantically relevant concepts. (b) During training, we jointly optimize the image encoder $f_{\theta}$, linear projection $W$, classifier $L_{\psi}$, and MLP $\mathcal{M}$ using both parametric and contrastive objectives, while refining the semantics of our unified representation via distillation of teacher cross-modal representations computed on the filtered dictionary. (c) The cross-modal covariance makes explicit which concept co-activations carry meaningful signal and should be preserved.
• Figure 3: Comparison of SimGCD (CLIP backbone) and SpectralGCD when using image features or cross-modal representations to train the classifier. Image features are slightly better on Old, while cross-modal ones improve on New.
• Figure 4: Accuracy vs. training time (s) for all methods on CUB. GET, TextGCD, and our method, require a preparation phase, whereas the unimodal approach SimGCD does not.
• Figure 5: Ablation on thresholds $\beta_e$ and $\beta_c$. For each threshold we report All accuracy and the resulting number of selected concepts. The black dashed line denotes performance without Spectral Filtering; the hatched bar marks the chosen configuration used for our main results.