InfoSculpt: Sculpting the Latent Space for Generalized Category Discovery

TL;DR

This work tackles Generalized Category Discovery (GCD) in open-world settings by reframing the problem through the Information Bottleneck (IB) lens and introducing InfoSculpt, which minimizes a dual Conditional Mutual Information (CMI) objective. By enforcing a Category-Level CMI on labeled data to compact known-class representations and an Instance-Level CMI on all data to distill augmentation-invariant features, InfoSculpt explicitly disentangles category-defining signals from instance-specific noise. The approach leverages a Vision Transformer backbone with contrastive learning and a carefully designed loss suite, achieving state-of-the-art results across 8 benchmarks and demonstrating strong generalization to novel categories. This information-theoretic latent-space sculpting offers a principled and scalable path toward robust open-world learning in vision tasks, with promising plug-and-play applicability to related GCD methods and potential extensions to more complex tasks.

Abstract

Generalized Category Discovery (GCD) aims to classify instances from both known and novel categories within a large-scale unlabeled dataset, a critical yet challenging task for real-world, open-world applications. However, existing methods often rely on pseudo-labeling, or two-stage clustering, which lack a principled mechanism to explicitly disentangle essential, category-defining signals from instance-specific noise. In this paper, we address this fundamental limitation by re-framing GCD from an information-theoretic perspective, grounded in the Information Bottleneck (IB) principle. We introduce InfoSculpt, a novel framework that systematically sculpts the representation space by minimizing a dual Conditional Mutual Information (CMI) objective. InfoSculpt uniquely combines a Category-Level CMI on labeled data to learn compact and discriminative representations for known classes, and a complementary Instance-Level CMI on all data to distill invariant features by compressing augmentation-induced noise. These two objectives work synergistically at different scales to produce a disentangled and robust latent space where categorical information is preserved while noisy, instance-specific details are discarded. Extensive experiments on 8 benchmarks demonstrate that InfoSculpt validating the effectiveness of our information-theoretic approach.

Figures (7)

• Figure 1: Generalized Category Discovery (GCD). Unlike traditional settings, GCD assumes the unlabeled data is a mixture of known and novel classes, which poses a core challenge for model learning.
• Figure 2: Architecture of InfoSculpt, which trains a feature encoder using a dual-level CMI loss—enhancing class separability and instance invariance—combined with contrastive objectives to sculpt a structured latent space tailored for GCD.
• Figure 3: Illustration of the IB principle. The network learns a compressed internal representation, Z that is maximally informative about the true label Y while discarding irrelevant information from X.
• Figure 4: Ablation study of InfoSculpt on four fine-grained datasets.
• Figure 5: Parameter sensitivity analysis for the target hardening parameter ($k$) and loss weights ($\lambda_{\text{cmi}}$, $\lambda_{\text{inst}}$, $\lambda_{\text{ent}}$). The evaluation is conducted on four fine-grained datasets: CUB-200-2011 (CUB), Stanford Cars (Cars), FGVC-Aircraft (Air.), and Herbarium19 (Her.). The dashed black line represents the mean accuracy. Yellow stars indicate the optimal value for each respective curve.