Classifying long legal documents using short random chunks

TL;DR

This paper tackles the challenge of classifying long legal documents by proposing a CPU-friendly architecture that uses 48 randomly sampled chunks (each ≤128 tokens) processed by DeBERTa V3 embeddings and an LSTM, enabling in-house deployment on multilingual data across 18 classes. The approach emphasizes robustness via epoch-wise random chunk sampling and a durable Temporal pipeline to orchestrate scalable, privacy-preserving workflows. Key results show a median weighted F-score of 0.898 and a CPU inference rate of about 4.98 seconds per document on CPU hardware, illustrating practical viability without large GPUs. The work contributes a deployable, end-to-end pipeline and a comprehensive deployment analysis, highlighting the trade-offs and potential for production use in legal-tech contexts.

Abstract

Classifying legal documents is a challenge, besides their specialized vocabulary, sometimes they can be very long. This means that feeding full documents to a Transformers-based models for classification might be impossible, expensive or slow. Thus, we present a legal document classifier based on DeBERTa V3 and a LSTM, that uses as input a collection of 48 randomly-selected short chunks (max 128 tokens). Besides, we present its deployment pipeline using Temporal, a durable execution solution, which allow us to have a reliable and robust processing workflow. The best model had a weighted F-score of 0.898, while the pipeline running on CPU had a processing median time of 498 seconds per 100 files.

Figures (3)

• Figure 1: Proposed architecture of the classifier.
• Figure 2: Temporal's pipeline architecture. Dashed lines are indirect communication managed by Temporal.
• Figure 3: Temporal's event history. Despite errors in the workflow, the pipeline continued until completed.