Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Identifying Legal Holdings with LLMs: A Systematic Study of Performance, Scale, and Memorization

Chuck Arvin

TL;DR

This study assesses how modern large language models (3B–90B+ parameters) perform on CaseHOLD, a benchmark for identifying legal holdings, and demonstrates a clear scaling effect where larger models achieve higher macro F1 scores without fine-tuning. Notably, GPT4o and AmazonNovaPro reach macro F1 scores of 0.744 and 0.720, competitive with specialized legal models, under zero-shot conditions. To address memorization concerns, the authors introduce a citation anonymization test which preserves semantics while altering case names and citations; results remain strong (macro F1 ≈ 0.728), suggesting performance is not merely memorized text. The work advances legal NLP benchmarking by showing promise for general-purpose LLMs in legal tasks and outlining future directions for memorization detection and broader evaluation across legal datasets.

Abstract

As large language models (LLMs) continue to advance in capabilities, it is essential to assess how they perform on established benchmarks. In this study, we present a suite of experiments to assess the performance of modern LLMs (ranging from 3B to 90B+ parameters) on CaseHOLD, a legal benchmark dataset for identifying case holdings. Our experiments demonstrate scaling effects - performance on this task improves with model size, with more capable models like GPT4o and AmazonNovaPro achieving macro F1 scores of 0.744 and 0.720 respectively. These scores are competitive with the best published results on this dataset, and do not require any technically sophisticated model training, fine-tuning or few-shot prompting. To ensure that these strong results are not due to memorization of judicial opinions contained in the training data, we develop and utilize a novel citation anonymization test that preserves semantic meaning while ensuring case names and citations are fictitious. Models maintain strong performance under these conditions (macro F1 of 0.728), suggesting the performance is not due to rote memorization. These findings demonstrate both the promise and current limitations of LLMs for legal tasks with important implications for the development and measurement of automated legal analytics and legal benchmarks.

Identifying Legal Holdings with LLMs: A Systematic Study of Performance, Scale, and Memorization

TL;DR

This study assesses how modern large language models (3B–90B+ parameters) perform on CaseHOLD, a benchmark for identifying legal holdings, and demonstrates a clear scaling effect where larger models achieve higher macro F1 scores without fine-tuning. Notably, GPT4o and AmazonNovaPro reach macro F1 scores of 0.744 and 0.720, competitive with specialized legal models, under zero-shot conditions. To address memorization concerns, the authors introduce a citation anonymization test which preserves semantics while altering case names and citations; results remain strong (macro F1 ≈ 0.728), suggesting performance is not merely memorized text. The work advances legal NLP benchmarking by showing promise for general-purpose LLMs in legal tasks and outlining future directions for memorization detection and broader evaluation across legal datasets.

Abstract

As large language models (LLMs) continue to advance in capabilities, it is essential to assess how they perform on established benchmarks. In this study, we present a suite of experiments to assess the performance of modern LLMs (ranging from 3B to 90B+ parameters) on CaseHOLD, a legal benchmark dataset for identifying case holdings. Our experiments demonstrate scaling effects - performance on this task improves with model size, with more capable models like GPT4o and AmazonNovaPro achieving macro F1 scores of 0.744 and 0.720 respectively. These scores are competitive with the best published results on this dataset, and do not require any technically sophisticated model training, fine-tuning or few-shot prompting. To ensure that these strong results are not due to memorization of judicial opinions contained in the training data, we develop and utilize a novel citation anonymization test that preserves semantic meaning while ensuring case names and citations are fictitious. Models maintain strong performance under these conditions (macro F1 of 0.728), suggesting the performance is not due to rote memorization. These findings demonstrate both the promise and current limitations of LLMs for legal tasks with important implications for the development and measurement of automated legal analytics and legal benchmarks.
Paper Structure (7 sections, 7 figures, 1 table)

This paper contains 7 sections, 7 figures, 1 table.

Table of Contents

  1. Introduction
  2. Research Design
  3. CaseHOLD Dataset
  4. Experimental Design
  5. Zero-Shot Empirical Results
  6. Have LLMs memorized the test set?
  7. Conclusion

Figures (7)

  • Figure 1: All models agree on the correct answer. As GPT4o states, "it directly addresses the Fourth Amendment scrutiny of searches and seizures on the high seas".
  • Figure 2: The models reach three distinct answers. As NovaPro states, "Given the context and the need for a holding that aligns with the discussion of service ordering and customer relationships, Option A is the most appropriate as it addresses a relevant doctrine in a manner consistent with the input text"
  • Figure 3: This prompt asks an LLM to read the citing text, analyze the options, and conclude with the best fitting option.
  • Figure 4: Match rate for answers between different models. LLMs frequently agree on the correct answer.
  • Figure 5: Macro F1 Scores on the CaseHOLD test set. Model performance improves with model size across all model families tested.
  • ...and 2 more figures