Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HiGen: Hierarchy-Aware Sequence Generation for Hierarchical Text Classification

Vidit Jain, Mukund Rungta, Yuchen Zhuang, Yue Yu, Zeyu Wang, Mu Gao, Jeffrey Skolnick, Chao Zhang

TL;DR

HiGen reframes hierarchical text classification as a sequence-generation task using a BART-based encoder–decoder to produce level-aware label sequences. It introduces a suite of losses, including a level-guided semantic loss, token constraint loss, and output-space loss, plus a task-specific pretraining regime with in-domain weak supervision to align text and label semantics across hierarchy levels. The approach achieves state-of-the-art Macro- and Micro-F1 on ENZYME, WOS, and NYT, with pronounced gains on long-tail and low-resource classes, while being notably more data- and time-efficient than baselines. The results demonstrate the value of dynamic, generation-based HTC with domain-adaptive pretraining for scalable, hierarchy-respecting text classification.

Abstract

Hierarchical text classification (HTC) is a complex subtask under multi-label text classification, characterized by a hierarchical label taxonomy and data imbalance. The best-performing models aim to learn a static representation by combining document and hierarchical label information. However, the relevance of document sections can vary based on the hierarchy level, necessitating a dynamic document representation. To address this, we propose HiGen, a text-generation-based framework utilizing language models to encode dynamic text representations. We introduce a level-guided loss function to capture the relationship between text and label name semantics. Our approach incorporates a task-specific pretraining strategy, adapting the language model to in-domain knowledge and significantly enhancing performance for classes with limited examples. Furthermore, we present a new and valuable dataset called ENZYME, designed for HTC, which comprises articles from PubMed with the goal of predicting Enzyme Commission (EC) numbers. Through extensive experiments on the ENZYME dataset and the widely recognized WOS and NYT datasets, our methodology demonstrates superior performance, surpassing existing approaches while efficiently handling data and mitigating class imbalance. The data and code will be released publicly.

HiGen: Hierarchy-Aware Sequence Generation for Hierarchical Text Classification

TL;DR

HiGen reframes hierarchical text classification as a sequence-generation task using a BART-based encoder–decoder to produce level-aware label sequences. It introduces a suite of losses, including a level-guided semantic loss, token constraint loss, and output-space loss, plus a task-specific pretraining regime with in-domain weak supervision to align text and label semantics across hierarchy levels. The approach achieves state-of-the-art Macro- and Micro-F1 on ENZYME, WOS, and NYT, with pronounced gains on long-tail and low-resource classes, while being notably more data- and time-efficient than baselines. The results demonstrate the value of dynamic, generation-based HTC with domain-adaptive pretraining for scalable, hierarchy-respecting text classification.

Abstract

Hierarchical text classification (HTC) is a complex subtask under multi-label text classification, characterized by a hierarchical label taxonomy and data imbalance. The best-performing models aim to learn a static representation by combining document and hierarchical label information. However, the relevance of document sections can vary based on the hierarchy level, necessitating a dynamic document representation. To address this, we propose HiGen, a text-generation-based framework utilizing language models to encode dynamic text representations. We introduce a level-guided loss function to capture the relationship between text and label name semantics. Our approach incorporates a task-specific pretraining strategy, adapting the language model to in-domain knowledge and significantly enhancing performance for classes with limited examples. Furthermore, we present a new and valuable dataset called ENZYME, designed for HTC, which comprises articles from PubMed with the goal of predicting Enzyme Commission (EC) numbers. Through extensive experiments on the ENZYME dataset and the widely recognized WOS and NYT datasets, our methodology demonstrates superior performance, surpassing existing approaches while efficiently handling data and mitigating class imbalance. The data and code will be released publicly.
Paper Structure (28 sections, 14 equations, 5 figures, 11 tables)

This paper contains 28 sections, 14 equations, 5 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Definition
  4. Methodology
  5. Pretraining Strategy
  6. Sequence to Sequence Modeling
  7. Training Objective
  8. Experiments
  9. Datasets
  10. Pretraining Datasets
  11. ENZYME Dataset
  12. Baselines
  13. Implementation Details
  14. Experimental Results
  15. Performance on Long-Tailed Distribution
  16. ...and 13 more sections

Figures (5)

  • Figure 1: BFS-based label flattening and random token & span masking employed during pretraining.
  • Figure 2: The proposed model's architecture consists of an encoder that takes the document and its corresponding label name as input. The decoder generates a hierarchical label output with 2 levels. To calculate the first three losses, the LM Head predicts a distribution over the vocabulary, and the hierarchy edges are considered. For the semantic loss, the text and label name representations from the encoder are projected onto a shared embedding space. Positive document and label semantics are pulled together, while negatives are pushed apart. The margins $\alpha_1$ and $\alpha_2$ control the attraction between levels 1 and 2, with $\alpha_1 > \alpha_2$.
  • Figure 3: Performance on ENZYME dataset for the long-tailed classes
  • Figure 4: Performance on ENZYME dataset on varying the training data proportion
  • Figure 5: ENZYME dataset statistics