Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Revisiting Transformer-based Models for Long Document Classification

Xiang Dai, Ilias Chalkidis, Sune Darkner, Desmond Elliott

TL;DR

This study interrogates how Transformer-based long-document classification can overcome the limitations of truncating long texts. By comparing sparse-attention (Longformer, BigBird) and hierarchical architectures across MIMIC-III, ECtHR, Hyperpartisan, and 20 News, it shows substantial performance gains when longer text is processed, with nuanced guidance on architectural choices. Task-adaptive pre-training and careful design of global attention, segment length, and overlap emerge as key levers for effectiveness and efficiency. The findings offer concrete recommendations for practitioners and demonstrate that transformers can outperform CNN baselines on genuinely long documents, reshaping approaches to real-world long-document classification tasks.

Abstract

The recent literature in text classification is biased towards short text sequences (e.g., sentences or paragraphs). In real-world applications, multi-page multi-paragraph documents are common and they cannot be efficiently encoded by vanilla Transformer-based models. We compare different Transformer-based Long Document Classification (TrLDC) approaches that aim to mitigate the computational overhead of vanilla transformers to encode much longer text, namely sparse attention and hierarchical encoding methods. We examine several aspects of sparse attention (e.g., size of local attention window, use of global attention) and hierarchical (e.g., document splitting strategy) transformers on four document classification datasets covering different domains. We observe a clear benefit from being able to process longer text, and, based on our results, we derive practical advice of applying Transformer-based models on long document classification tasks.

Revisiting Transformer-based Models for Long Document Classification

TL;DR

This study interrogates how Transformer-based long-document classification can overcome the limitations of truncating long texts. By comparing sparse-attention (Longformer, BigBird) and hierarchical architectures across MIMIC-III, ECtHR, Hyperpartisan, and 20 News, it shows substantial performance gains when longer text is processed, with nuanced guidance on architectural choices. Task-adaptive pre-training and careful design of global attention, segment length, and overlap emerge as key levers for effectiveness and efficiency. The findings offer concrete recommendations for practitioners and demonstrate that transformers can outperform CNN baselines on genuinely long documents, reshaping approaches to real-world long-document classification tasks.

Abstract

The recent literature in text classification is biased towards short text sequences (e.g., sentences or paragraphs). In real-world applications, multi-page multi-paragraph documents are common and they cannot be efficiently encoded by vanilla Transformer-based models. We compare different Transformer-based Long Document Classification (TrLDC) approaches that aim to mitigate the computational overhead of vanilla transformers to encode much longer text, namely sparse attention and hierarchical encoding methods. We examine several aspects of sparse attention (e.g., size of local attention window, use of global attention) and hierarchical (e.g., document splitting strategy) transformers on four document classification datasets covering different domains. We observe a clear benefit from being able to process longer text, and, based on our results, we derive practical advice of applying Transformer-based models on long document classification tasks.
Paper Structure (42 sections, 3 equations, 9 figures, 16 tables)

This paper contains 42 sections, 3 equations, 9 figures, 16 tables.

Table of Contents

  1. Introduction
  2. Problem Formulation and Datasets
  3. Approaches
  4. Sparse-Attention Transformers
  5. Hierarchical Transformers
  6. Experimental Setup
  7. Backbone Models
  8. Evaluation metrics
  9. Experiments
  10. Task-adaptive pre-training is a promising first step.
  11. Longformer
  12. Small local attention windows are effective and efficient.
  13. Considering a small number of tokens for global attention improves the stability of the training process.
  14. Equally distributing global tokens across the sequence is better than content-based attribution.
  15. Hierarchical Transformers
  16. ...and 27 more sections

Figures (9)

  • Figure 1: The effectiveness of Longformer, a long-document Transformer, on the MIMIC-III development set. There is a clear benefit from being able to process longer text.
  • Figure 2: The distribution of document lengths. A log-$10$ scale is used for the X axis.
  • Figure 3: A comparison of three types of attention operations. The example sequence contains 7 tokens; we set local attention window size as 2, and only the first token using global attention. Note that these curves are bi-directional that tokens can attend to each other.
  • Figure 4: A high-level illustration of hierarchical Transformers. A shared pre-trained RoBERTa is used to encode each segment, and a two layer transformer blocks is used to capture the interaction between different segments. Finally, contextual segment representations are aggregated into a document representation.
  • Figure 5: Task-adaptive pre-training (right side in each plot) can improve the effectiveness (measured on the development sets) of pre-trained models by a large margin on MIMIC-III, but small on ECtHR. $\Delta$: the difference between mean values of compared experiments.
  • ...and 4 more figures