Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Transformer-based Autoregressive Decoder Architecture for Hierarchical Text Classification

Younes Yousef, Lukas Galke, Ansgar Scherp

TL;DR

This work introduces RA-DAr, a transformer-based autoregressive decoder architecture for hierarchical text classification that relies on a RoBERTa encoder and symbolic label sequences, removing the need for a graph encoder or label semantics. Evaluated on three HTC benchmarks, RA-DAr achieves results competitive with state-of-the-art methods while substantially reducing training and inference time. The authors demonstrate that ordering label sequences from children to parents yields consistent gains, and that batch-level focal loss helps especially on harder cases, with minimal reliance on label semantics. This approach offers practical benefits in speed and flexibility, enabling easier component replacement and deployment in real-world HTC tasks.

Abstract

Recent approaches in hierarchical text classification (HTC) rely on the capabilities of a pre-trained transformer model and exploit the label semantics and a graph encoder for the label hierarchy. In this paper, we introduce an effective hierarchical text classifier RADAr (Transformer-based Autoregressive Decoder Architecture) that is based only on an off-the-shelf RoBERTa transformer to process the input and a custom autoregressive decoder with two decoder layers for generating the classification output. Thus, unlike existing approaches for HTC, the encoder of RADAr has no explicit encoding of the label hierarchy and the decoder solely relies on the label sequences of the samples observed during training. We demonstrate on three benchmark datasets that RADAr achieves results competitive to the state of the art with less training and inference time. Our model consistently performs better when organizing the label sequences from children to parents versus the inverse, as done in existing HTC approaches. Our experiments show that neither the label semantics nor an explicit graph encoder for the hierarchy is needed. This has strong practical implications for HTC as the architecture has fewer requirements and provides a speed-up by a factor of 2 at inference time. Moreover, training a separate decoder from scratch in conjunction with fine-tuning the encoder allows future researchers and practitioners to exchange the encoder part as new models arise. The source code is available at https://github.com/yousef-younes/RADAr.

A Transformer-based Autoregressive Decoder Architecture for Hierarchical Text Classification

TL;DR

This work introduces RA-DAr, a transformer-based autoregressive decoder architecture for hierarchical text classification that relies on a RoBERTa encoder and symbolic label sequences, removing the need for a graph encoder or label semantics. Evaluated on three HTC benchmarks, RA-DAr achieves results competitive with state-of-the-art methods while substantially reducing training and inference time. The authors demonstrate that ordering label sequences from children to parents yields consistent gains, and that batch-level focal loss helps especially on harder cases, with minimal reliance on label semantics. This approach offers practical benefits in speed and flexibility, enabling easier component replacement and deployment in real-world HTC tasks.

Abstract

Recent approaches in hierarchical text classification (HTC) rely on the capabilities of a pre-trained transformer model and exploit the label semantics and a graph encoder for the label hierarchy. In this paper, we introduce an effective hierarchical text classifier RADAr (Transformer-based Autoregressive Decoder Architecture) that is based only on an off-the-shelf RoBERTa transformer to process the input and a custom autoregressive decoder with two decoder layers for generating the classification output. Thus, unlike existing approaches for HTC, the encoder of RADAr has no explicit encoding of the label hierarchy and the decoder solely relies on the label sequences of the samples observed during training. We demonstrate on three benchmark datasets that RADAr achieves results competitive to the state of the art with less training and inference time. Our model consistently performs better when organizing the label sequences from children to parents versus the inverse, as done in existing HTC approaches. Our experiments show that neither the label semantics nor an explicit graph encoder for the hierarchy is needed. This has strong practical implications for HTC as the architecture has fewer requirements and provides a speed-up by a factor of 2 at inference time. Moreover, training a separate decoder from scratch in conjunction with fine-tuning the encoder allows future researchers and practitioners to exchange the encoder part as new models arise. The source code is available at https://github.com/yousef-younes/RADAr.
Paper Structure (19 sections, 2 figures, 4 tables, 1 algorithm)

This paper contains 19 sections, 2 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Related Work
  2. Encoder-only Models
  3. Encoder-Decoder Models
  4. The RA-DAr Model
  5. Encoder
  6. Decoder
  7. Training and Testing
  8. Experimental Apparatus
  9. Datasets and Metrics
  10. Data Preprocessing
  11. Procedure and Hyperparameters
  12. Results
  13. Discussion
  14. Ablation Study
  15. Error Analysis
  16. ...and 4 more sections

Figures (2)

  • Figure 1: The RA-DAr Model Architecture
  • Figure 2: Label preprocessing and tokenization. Line a) contains the original labels. Line b) maps the original labels to the symbolic labels. Line c) adds the level separator token <unk>. Line d) organizes the labels level-wise from children to parents. Line e) contains the padded tokenizer output.