Modeling Text-Label Alignment for Hierarchical Text Classification

TL;DR

The Hierarchical Text-Label Alignment model is introduced, which leverages BERT as the text encoder and GPTrans as the graph encoder and integrates text-label embeddings to generate hierarchy-aware representations.

Abstract

Hierarchical Text Classification (HTC) aims to categorize text data based on a structured label hierarchy, resulting in predicted labels forming a sub-hierarchy tree. The semantics of the text should align with the semantics of the labels in this sub-hierarchy. With the sub-hierarchy changing for each sample, the dynamic nature of text-label alignment poses challenges for existing methods, which typically process text and labels independently. To overcome this limitation, we propose a Text-Label Alignment (TLA) loss specifically designed to model the alignment between text and labels. We obtain a set of negative labels for a given text and its positive label set. By leveraging contrastive learning, the TLA loss pulls the text closer to its positive label and pushes it away from its negative label in the embedding space. This process aligns text representations with related labels while distancing them from unrelated ones. Building upon this framework, we introduce the Hierarchical Text-Label Alignment (HTLA) model, which leverages BERT as the text encoder and GPTrans as the graph encoder and integrates text-label embeddings to generate hierarchy-aware representations. Experimental results on benchmark datasets and comparison with existing baselines demonstrate the effectiveness of HTLA for HTC.

Figures (4)

• Figure 1: Architecture of the Hierarchical Text-Label Alignment (HTLA) model. For a label $i$, its feature is combined with the text feature $h_{text}$ through addition to produce the composite feature $C_i \in \mathbb{R}^{d_h}$ for each label. A shared classifier is then utilized for each $C_i$, and the corresponding logit $l_i$ is selected from the output vector. The model is jointly optimized for BCE and TLA loss.
• Figure 2: Model performance across label prevalence categories
• Figure 3: Model performance across hierarchy levels
• Figure 4: Model performance across label paths