Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Lawma: The Power of Specialization for Legal Annotation

Ricardo Dominguez-Olmedo, Vedant Nanda, Rediet Abebe, Stefan Bechtold, Christoph Engel, Jens Frankenreiter, Krishna Gummadi, Moritz Hardt, Michael Livermore

TL;DR

The paper tackles the high cost of legal text annotation and introduces CaselawQA, a 260-task benchmark built from SCDB and USCAD to systematically evaluate ML annotation for legal data. It challenges the predominance of large commercial LLM prompting by showing that small, lightly fine-tuned open-weight models (Lawma) achieve higher accuracy with far fewer labeled examples. The results reveal strong sample efficiency, with hundreds to about a thousand labeled examples sufficing, and demonstrate broad generalization to unseen databases when tasks are precisely specialized. The work supports building an ecosystem of fine-tuned, task-specific models as a practical alternative to prompting large generalist models in empirical legal studies. CaselawQA is positioned as a benchmark for future model development and evaluation in the legal domain.

Abstract

Annotation and classification of legal text are central components of empirical legal research. Traditionally, these tasks are often delegated to trained research assistants. Motivated by the advances in language modeling, empirical legal scholars are increasingly turning to prompting commercial models, hoping that it will alleviate the significant cost of human annotation. Despite growing use, our understanding of how to best utilize large language models for legal annotation remains limited. To bridge this gap, we introduce CaselawQA, a benchmark comprising 260 legal annotation tasks, nearly all new to the machine learning community. We demonstrate that commercial models, such as GPT-4.5 and Claude 3.7 Sonnet, achieve non-trivial yet highly variable accuracy, generally falling short of the performance required for legal work. We then demonstrate that small, lightly fine-tuned models outperform commercial models. A few hundred to a thousand labeled examples are usually enough to achieve higher accuracy. Our work points to a viable alternative to the predominant practice of prompting commercial models. For concrete legal annotation tasks with some available labeled data, researchers are likely better off using a fine-tuned open-source model.

Lawma: The Power of Specialization for Legal Annotation

TL;DR

The paper tackles the high cost of legal text annotation and introduces CaselawQA, a 260-task benchmark built from SCDB and USCAD to systematically evaluate ML annotation for legal data. It challenges the predominance of large commercial LLM prompting by showing that small, lightly fine-tuned open-weight models (Lawma) achieve higher accuracy with far fewer labeled examples. The results reveal strong sample efficiency, with hundreds to about a thousand labeled examples sufficing, and demonstrate broad generalization to unseen databases when tasks are precisely specialized. The work supports building an ecosystem of fine-tuned, task-specific models as a practical alternative to prompting large generalist models in empirical legal studies. CaselawQA is positioned as a benchmark for future model development and evaluation in the legal domain.

Abstract

Annotation and classification of legal text are central components of empirical legal research. Traditionally, these tasks are often delegated to trained research assistants. Motivated by the advances in language modeling, empirical legal scholars are increasingly turning to prompting commercial models, hoping that it will alleviate the significant cost of human annotation. Despite growing use, our understanding of how to best utilize large language models for legal annotation remains limited. To bridge this gap, we introduce CaselawQA, a benchmark comprising 260 legal annotation tasks, nearly all new to the machine learning community. We demonstrate that commercial models, such as GPT-4.5 and Claude 3.7 Sonnet, achieve non-trivial yet highly variable accuracy, generally falling short of the performance required for legal work. We then demonstrate that small, lightly fine-tuned models outperform commercial models. A few hundred to a thousand labeled examples are usually enough to achieve higher accuracy. Our work points to a viable alternative to the predominant practice of prompting commercial models. For concrete legal annotation tasks with some available labeled data, researchers are likely better off using a fine-tuned open-source model.
Paper Structure (35 sections, 16 figures, 1 table)

This paper contains 35 sections, 16 figures, 1 table.

Table of Contents

  1. Introduction
  2. Our contributions
  3. Related work
  4. Adoption of large language models in the legal community.
  5. Benchmarks for legal tasks.
  6. Large language models for the legal domain.
  7. Data annotation and labeling.
  8. Limitations
  9. CaselawQA: a benchmark for legal annotation
  10. Data sources
  11. Construction of the classification tasks
  12. Evaluation methodology
  13. Evaluation baselines
  14. Few-shot and Chain of Thought prompting.
  15. Language models for the legal domain.
  16. ...and 20 more sections

Figures (16)

  • Figure 1: The power of specialization: Performance of various language models on the CaselawQA benchmark for legal annotation. The Lawma models, specialized for legal annotation, outperform all other models.
  • Figure 2: General statistics of the court opinions and legal classification tasks considered.
  • Figure 3: Example task and prompt corresponding to the Supreme Court "precedent alteration" variable.
  • Figure 4: Distribution of task performance across all tasks for Llama 3.1 405B Instruct and Lawma 8B.
  • Figure 5: Accuracy of models on the ten highlighted legal classification tasks.
  • ...and 11 more figures