GLEAN: Generalized Category Discovery with Diverse and Quality-Enhanced LLM Feedback

TL;DR

GLEAN tackles open-world generalized category discovery by integrating three distinct LLM feedback types into a unified learning framework. It combines neighborhood contrastive learning on similar instances, category-characterized descriptions, and pseudo category alignment to refine representations and semantically label novel clusters, all guided by a joint loss L = L^{ce} + L^{ncl} + λ L^{align}. Across BANKING, CLINC, and StackOverflow, GLEAN achieves state-of-the-art results, particularly under limited known-category ratios, and ablation shows that both feedback diversity and quality critically drive performance. The approach reduces reliance on expensive human annotations while demonstrating that open-source LLMs can provide competitive, scalable supervision for open-world text classification tasks.

Abstract

Generalized Category Discovery (GCD) is a practical and challenging open-world task that aims to recognize both known and novel categories in unlabeled data using limited labeled data from known categories. Due to the lack of supervision, previous GCD methods face significant challenges, such as difficulty in rectifying errors for confusing instances, and inability to effectively uncover and leverage the semantic meanings of discovered clusters. Therefore, additional annotations are usually required for real-world applicability. However, human annotation is extremely costly and inefficient. To address these issues, we propose GLEAN, a unified framework for generalized category discovery that actively learns from diverse and quality-enhanced LLM feedback. Our approach leverages three different types of LLM feedback to: (1) improve instance-level contrastive features, (2) generate category descriptions, and (3) align uncertain instances with LLM-selected category descriptions. Extensive experiments demonstrate the superior performance of \MethodName over state-of-the-art models across diverse datasets, metrics, and supervision settings. Our code is available at https://github.com/amazon-science/Glean.

Figures (8)

• Figure 1: Generalized Category Discovery aims to automatically categorize unlabeled data by leveraging the information from a limited number of labeled data from known categories, while the unlabeled data may come from both known and novel categories.
• Figure 2: Pipeline of GLEAN. Both labeled and unlabeled data are first forwarded to a text encoder/backbone to extract features for k-means clustering. Then we compute entropy and select instances with high entropy as ambiguous data to obtain LLM feedback for further refinement. Specifically, we query LLM to (1) select similar instances, (2) generate category descriptions and (3) assign pseudo categories to ambiguous data. Lastly, the three diverse feedback types are leveraged for model training via neighborhood contrastive learning and pseudo category alignment. During inference, we only utilize the text encoder and obtain final results via K-Means clustering on the extracted features. Illustration of the three types of LLM feedback with concrete examples is provided in Figure \ref{['fig:llm_feedback']}.
• Figure 3: Illustration of three different types of LLM feedback utilized in GLEAN. Illustration of the whole pipeline is provided in Figure \ref{['fig:pipeline_illustration']}.
• Figure 4: Influence of the number of query samples. Increasing the number of query samples generally leads to better performance, especially on the CLINC dataset. The performance gain starts to saturate on the BANKING dataset as the number of query samples reaches 500, we hypothesize that this is because the BANKING dataset is more challenging and distinguishing ambiguous samples and categories becomes increasingly difficult.
• Figure 5: Implementation of the four evaluation metrics for category characterization: Coverage Score, Uniformity Score, SeMatching Score and Informative Score.