Can Large Language Models Serve as Effective Classifiers for Hierarchical Multi-Label Classification of Scientific Documents at Industrial Scale?

TL;DR

This work tackles scalable hierarchical multi-label classification of scientific documents under dynamic taxonomies by combining Large Language Models with dense retrieval to avoid retraining. A bi-encoder–assisted LLM framework prunes large label sets, while several LLM-based strategies (notably LLM-SelectP) perform label selection with high accuracy (0.943) and strong precision across a dynamic SSRN taxonomy. The study introduces a tailored evaluation framework using SME feedback via GENEX, demonstrates substantial cost savings (from $3.50 to $0.20 per document), and provides public code and a dataset description framework to support real-world deployment. The results indicate that LLMs, when properly integrated with retrieval components, can effectively manage industrial-scale, evolving label spaces with improved efficiency and consistency.

Abstract

We address the task of hierarchical multi-label classification (HMC) of scientific documents at an industrial scale, where hundreds of thousands of documents must be classified across thousands of dynamic labels. The rapid growth of scientific publications necessitates scalable and efficient methods for classification, further complicated by the evolving nature of taxonomies--where new categories are introduced, existing ones are merged, and outdated ones are deprecated. Traditional machine learning approaches, which require costly retraining with each taxonomy update, become impractical due to the high overhead of labelled data collection and model adaptation. Large Language Models (LLMs) have demonstrated great potential in complex tasks such as multi-label classification. However, applying them to large and dynamic taxonomies presents unique challenges as the vast number of labels can exceed LLMs' input limits. In this paper, we present novel methods that combine the strengths of LLMs with dense retrieval techniques to overcome these challenges. Our approach avoids retraining by leveraging zero-shot HMC for real-time label assignment. We evaluate the effectiveness of our methods on SSRN, a large repository of preprints spanning multiple disciplines, and demonstrate significant improvements in both classification accuracy and cost-efficiency. By developing a tailored evaluation framework for dynamic taxonomies and publicly releasing our code, this research provides critical insights into applying LLMs for document classification, where the number of classes corresponds to the number of nodes in a large taxonomy, at an industrial scale.

Figures (11)

• Figure 1: An illustration of our most effective proposed method, LLM-Select-Pointwise (LLM-SelectP).
• Figure 2: Comparative performance of different bi-encoders.
• Figure 3: A document can belong to multiple perfect sets, each consisting of different combinations of relevant labels.
• Figure 4: The prompt of description generation.
• Figure 5: The prompt of LLM-Traverse-LLM-Select (TravSelect) method.