SelEx: Self-Expertise in Fine-Grained Generalized Category Discovery

TL;DR

This paper introduces a novel concept called `self-expertise', which enhances the model's ability to recognize subtle differences and uncover unknown categories, and combines unsupervised and supervised self-expertise strategies to refine the model's discernment and generalization.

Abstract

In this paper, we address Generalized Category Discovery, aiming to simultaneously uncover novel categories and accurately classify known ones. Traditional methods, which lean heavily on self-supervision and contrastive learning, often fall short when distinguishing between fine-grained categories. To address this, we introduce a novel concept called `self-expertise', which enhances the model's ability to recognize subtle differences and uncover unknown categories. Our approach combines unsupervised and supervised self-expertise strategies to refine the model's discernment and generalization. Initially, hierarchical pseudo-labeling is used to provide `soft supervision', improving the effectiveness of self-expertise. Our supervised technique differs from traditional methods by utilizing more abstract positive and negative samples, aiding in the formation of clusters that can generalize to novel categories. Meanwhile, our unsupervised strategy encourages the model to sharpen its category distinctions by considering within-category examples as `hard' negatives. Supported by theoretical insights, our empirical results showcase that our method outperforms existing state-of-the-art techniques in Generalized Category Discovery across several fine-grained datasets. Our code is available at: https://github.com/SarahRastegar/SelEx.

Figures (21)

• Figure 1: The motivation for self-expertise(a) Unsupervised Contrastive Learning. shows identical repulsion for both misclassified cat and lion. (b) Unsupervised Self-Expertise. focuses on distinguishing hard negatives within a category cluster, applying less repulsion to external members, and varying the repulsion for misclassified samples, resulting in milder repulsion for the mislabeled cat and stronger for the lion. (c) Supervised Contrastive Learning. focuses on attracting similar category members, leaving others unaffected. (d) Supervised Self-Expertise. graduates the attraction of samples based on semantic similarity. Both a misclassified cat and lion are equally attracted to the cat sample at the feline level. Supervised and unsupervised self-expertise together enhance accuracy by attracting similar samples and repelling dissimilar ones.
• Figure 2: Bayesian Network for the Generalized Category Discovery. Shaded nodes are observed variables $\mathbf{x_i}$, $\mathbf{x_j}$ corresponds to images $i$ and $j$, and $\mathbf{c_i}$ and $\mathbf{c_j}$ which are the ground-truth category variable. $z_j$ is the latent category variable extracted from the model.
• Figure 3: Self-expertise for Generalized Category Discovery. Our method integrates three key components. The initial component is the Hierarchical Semi-Supervised K-means, which extracts pseudo-labels across multiple levels of expertise. Utilizing these pseudo-labels, the second component adopts unsupervised self-expertise by identifying hard negatives within each pseudo-label for enhanced differentiation across expertise tiers. The final component applies supervised self-expertise, recognizing samples sharing the same pseudo-label as weak positives to boost positive sample frequency while employing external pseudo-labels as strong negatives. This strategy accelerates cluster formation by capturing abstract-level similarities.
• Figure 4: Illustrating the distinction in target matrix formulation for unsupervised self-expertise(a) Unsupervised Contrastive Learning. Each sample's augmented version is deemed positive, with all other samples marked as negative. (b) Unsupervised Self-Expertise. On the contrary, our method dynamically adjusts the negativity weight of each sample according to semantic similarity, treating categories with higher similarity (e.g., within the 'cat' category) as more strict negatives. Conversely, semantically different categories (e.g., 'lion' vs. 'cat') incorporate a degree of uncertainty in their negativity, quantified as $\frac{1}{2}$ to reflect the semantic differences between negatives. Since the target matrix represents probabilities, normalization is required to ensure validity.
• Figure 5: Supervised self-expertise. Similar to playing a game of twenty questions, our method employs supervised self-expertise to discern sample attributes. As the process evolves, the attributes it discerns between are increasingly specific. Focusing on the Grosbeak classification, the model initially utilizes the leftmost segment of its latent representation (indicated by the yellow square) to ascertain whether a sample is seen. The model then allocates its representation's yellow and blue square parts to identify whether the subject is a jungle bird. Subsequently, it dedicates the latter part (represented by the green rectangle) to determine if this jungle bird is a Grosbeak.

Theorems & Definitions (4)

• theorem 1
• theorem 2
• theorem 3
• proof