Learning Part Knowledge to Facilitate Category Understanding for Fine-Grained Generalized Category Discovery

TL;DR

PartGCD addresses the brittleness of fine-grained generalized category discovery by integrating explicit part knowledge into learning. It introduces Adaptive Part Decomposition to automatically extract class-specific parts and Part Discrepancy Regularization to encourage distinct part representations, combined with a calibrated prototype and GMM-based part modeling. The approach yields state-of-the-art results on multiple fine-grained benchmarks while remaining competitive on generic datasets, validating the importance of local, part-level cues for robust category understanding in unlabeled data. The work also provides theoretical grounding for Sinkhorn-based prototype calibration and comprehensive analyses, highlighting practical benefits in real-world-like, fine-grained discovery tasks.

Abstract

Generalized Category Discovery (GCD) aims to classify unlabeled data containing both seen and novel categories. Although existing methods perform well on generic datasets, they struggle in fine-grained scenarios. We attribute this difficulty to their reliance on contrastive learning over global image features to automatically capture discriminative cues, which fails to capture the subtle local differences essential for distinguishing fine-grained categories. Therefore, in this paper, we propose incorporating part knowledge to address fine-grained GCD, which introduces two key challenges: the absence of annotations for novel classes complicates the extraction of the part features, and global contrastive learning prioritizes holistic feature invariance, inadvertently suppressing discriminative local part patterns. To address these challenges, we propose PartGCD, including 1) Adaptive Part Decomposition, which automatically extracts class-specific semantic parts via Gaussian Mixture Models, and 2) Part Discrepancy Regularization, enforcing explicit separation between part features to amplify fine-grained local part distinctions. Experiments demonstrate state-of-the-art performance across multiple fine-grained benchmarks while maintaining competitiveness on generic datasets, validating the effectiveness and robustness of our approach.

Figures (12)

• Figure 1: Previous methods rely on global features, leading to inaccurate discrimination cues. Our PartGCD captures distinct part information, achieving accurate and discriminative perception.
• Figure 2: The framework of our PartGCD. At the beginning of each epoch, we select candidate samples for each class via calibrated prototypes and then use these samples to construct part-based GMMs, assigning each sample corresponding GMM parameters. At each iteration, we generate part attention maps to guide part knowledge learning.
• Figure 3: (a) the mean similarity of patch features for each image averaged across the dataset throughout training. (b) the distribution of part features of a random class
• Figure 4: The sensitivity to $K$ values (left) and the silhouette coefficient (right) on the SCars dataset.
• Figure 5: Impact of hyper-parameters.