Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

No Argument Left Behind: Overlapping Chunks for Faster Processing of Arbitrarily Long Legal Texts

Israel Fama, Bárbara Bueno, Alexandre Alcoforado, Thomas Palmeira Ferraz, Arnold Moya, Anna Helena Reali Costa

TL;DR

This work tackles the challenge of analyzing arbitrarily long legal documents for Legal Judgment Prediction in the Brazilian context. It introduces uBERT, a Transformer-RNN hybrid that processes full texts by splitting them into overlapping chunks, extracting chunk representations from the final Transformer layers, and integrating them with an RNN. Results show that overlapive chunking yields improvements over BERT+LSTM and that uBERT generally beats ULMFiT on many long-text scenarios while remaining significantly faster; however, ULMFiT can still outperform uBERT on the very longest texts, highlighting a trade-off between accuracy and efficiency. Practically, uBERT offers a scalable approach for full-document analysis in legal NLP and motivates further work on chunking strategies and cross-language validation.

Abstract

In a context where the Brazilian judiciary system, the largest in the world, faces a crisis due to the slow processing of millions of cases, it becomes imperative to develop efficient methods for analyzing legal texts. We introduce uBERT, a hybrid model that combines Transformer and Recurrent Neural Network architectures to effectively handle long legal texts. Our approach processes the full text regardless of its length while maintaining reasonable computational overhead. Our experiments demonstrate that uBERT achieves superior performance compared to BERT+LSTM when overlapping input is used and is significantly faster than ULMFiT for processing long legal documents.

No Argument Left Behind: Overlapping Chunks for Faster Processing of Arbitrarily Long Legal Texts

TL;DR

This work tackles the challenge of analyzing arbitrarily long legal documents for Legal Judgment Prediction in the Brazilian context. It introduces uBERT, a Transformer-RNN hybrid that processes full texts by splitting them into overlapping chunks, extracting chunk representations from the final Transformer layers, and integrating them with an RNN. Results show that overlapive chunking yields improvements over BERT+LSTM and that uBERT generally beats ULMFiT on many long-text scenarios while remaining significantly faster; however, ULMFiT can still outperform uBERT on the very longest texts, highlighting a trade-off between accuracy and efficiency. Practically, uBERT offers a scalable approach for full-document analysis in legal NLP and motivates further work on chunking strategies and cross-language validation.

Abstract

In a context where the Brazilian judiciary system, the largest in the world, faces a crisis due to the slow processing of millions of cases, it becomes imperative to develop efficient methods for analyzing legal texts. We introduce uBERT, a hybrid model that combines Transformer and Recurrent Neural Network architectures to effectively handle long legal texts. Our approach processes the full text regardless of its length while maintaining reasonable computational overhead. Our experiments demonstrate that uBERT achieves superior performance compared to BERT+LSTM when overlapping input is used and is significantly faster than ULMFiT for processing long legal documents.

Paper Structure

This paper contains 16 sections, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Transformer-Based Approaches for Long Text Processing in Legal NLP
  4. Critiques and Limitations in Legal NLP Research
  5. Proposal
  6. Experiments
  7. Data
  8. Models
  9. Baselines
  10. Computational Infrastructure and Resources
  11. Evaluation Metrics
  12. Results
  13. Processing the Full Text Requires Overlap
  14. uBERT with Overlap is still Significantly Faster than ULMFiT
  15. ULMFiT Outperforms uBERT on Longer Texts
  16. ...and 1 more sections

Figures (4)

  • Figure 1: uBERT architecture.
  • Figure 2: Overlapping chunks example.
  • Figure 3: Macro-F1 score x Avg. Tokens/Group across different groups of same size ranked by the length. The error bars represent 95% confidence intervals obtained with bootstrap resampling.
  • Figure 4: Critical difference diagram showing pairwise statistical comparison between baselines and varying overlap sizes for uBERT using the MCC. Connecting bars represent no statistical difference between methods.