When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset

TL;DR

This work investigates when domain-specific pretraining helps in legal NLP by introducing CaseHOLD, a large, challenging dataset of 53k+ multiple-choice prompts that require identifying the legal holding of cited cases. It systematically compares a BiLSTM baseline, general-domain BERT, and two Legal-BERT variants (one pretrained on legal text with a standard vocabulary and one with a custom legal vocabulary), showing that substantial gains occur on CaseHOLD with domain-specific pretraining (a 7.2 percentage-point increase in F1, about 12% relative to BERT), while gains on easier legal tasks are smaller. The authors introduce a Domain Specificity Score (DS) to quantify task similarity to the pretraining corpus and demonstrate that higher DS tasks (like CaseHOLD) benefit more from domain pretraining, whereas easier, lower-DS tasks show limited improvements. The study provides practical guidance on when to invest in resource-intensive pretraining, suggesting that gains scale with task difficulty and domain similarity, and it contributes publicly available datasets and models to advance legal NLP benchmarks and analysis.

Abstract

While self-supervised learning has made rapid advances in natural language processing, it remains unclear when researchers should engage in resource-intensive domain-specific pretraining (domain pretraining). The law, puzzlingly, has yielded few documented instances of substantial gains to domain pretraining in spite of the fact that legal language is widely seen to be unique. We hypothesize that these existing results stem from the fact that existing legal NLP tasks are too easy and fail to meet conditions for when domain pretraining can help. To address this, we first present CaseHOLD (Case Holdings On Legal Decisions), a new dataset comprised of over 53,000+ multiple choice questions to identify the relevant holding of a cited case. This dataset presents a fundamental task to lawyers and is both legally meaningful and difficult from an NLP perspective (F1 of 0.4 with a BiLSTM baseline). Second, we assess performance gains on CaseHOLD and existing legal NLP datasets. While a Transformer architecture (BERT) pretrained on a general corpus (Google Books and Wikipedia) improves performance, domain pretraining (using corpus of approximately 3.5M decisions across all courts in the U.S. that is larger than BERT's) with a custom legal vocabulary exhibits the most substantial performance gains with CaseHOLD (gain of 7.2% on F1, representing a 12% improvement on BERT) and consistent performance gains across two other legal tasks. Third, we show that domain pretraining may be warranted when the task exhibits sufficient similarity to the pretraining corpus: the level of performance increase in three legal tasks was directly tied to the domain specificity of the task. Our findings inform when researchers should engage resource-intensive pretraining and show that Transformer-based architectures, too, learn embeddings suggestive of distinct legal language.

Figures (3)

• Figure 1: Mean macro F1 scores over 3 folds, with $\pm 1.96 \times \text{standard error}$, for train volume variant.
• Figure 2: Mean macro F1 scores over 3 folds, with $\pm1.96 \times \textbf{standard error}$, for prompt difficulty variant.
• Figure 3: Density histograms of DS scores of examples for Terms of Service, CaseHOLD, and both tasks.