Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CaseSumm: A Large-Scale Dataset for Long-Context Summarization from U.S. Supreme Court Opinions

Mourad Heddaya, Kyle MacMillan, Anup Malani, Hongyuan Mei, Chenhao Tan

TL;DR

CaseSumm introduces a large-scale long-context legal summarization dataset consisting of 25.6k U.S. Supreme Court opinions and their official syllabuses (1815–2019), enabling evaluation of long-form legal summaries. It provides a comprehensive, ground-truth–driven benchmark by comparing zero-shot and instruction-tuned LLMs (GPT-4 Turbo and Mistral 7b Instruct with LoRA) against official syllabuses and public controls (Oyez, Westlaw) using automatic metrics and expert human evaluation. The study reveals notable mismatches between automatic metrics and human judgments, with GPT-4 summaries often preferred by humans for clarity and accuracy despite lower automatic scores, and fine-tuned Mistral achieving strong automatic performance but greater factual errors. It also shows that LLM-based evaluation tools like G-Eval do not consistently align better with human judgments than traditional metrics, underscoring the continued need for human evaluation in high-stakes domains. CaseSumm is publicly available on HuggingFace and offers a resource to advance long-context legal summarization research and evaluation methodology.

Abstract

This paper introduces CaseSumm, a novel dataset for long-context summarization in the legal domain that addresses the need for longer and more complex datasets for summarization evaluation. We collect 25.6K U.S. Supreme Court (SCOTUS) opinions and their official summaries, known as "syllabuses." Our dataset is the largest open legal case summarization dataset, and is the first to include summaries of SCOTUS decisions dating back to 1815. We also present a comprehensive evaluation of LLM-generated summaries using both automatic metrics and expert human evaluation, revealing discrepancies between these assessment methods. Our evaluation shows Mistral 7b, a smaller open-source model, outperforms larger models on most automatic metrics and successfully generates syllabus-like summaries. In contrast, human expert annotators indicate that Mistral summaries contain hallucinations. The annotators consistently rank GPT-4 summaries as clearer and exhibiting greater sensitivity and specificity. Further, we find that LLM-based evaluations are not more correlated with human evaluations than traditional automatic metrics. Furthermore, our analysis identifies specific hallucinations in generated summaries, including precedent citation errors and misrepresentations of case facts. These findings demonstrate the limitations of current automatic evaluation methods for legal summarization and highlight the critical role of human evaluation in assessing summary quality, particularly in complex, high-stakes domains. CaseSumm is available at https://huggingface.co/datasets/ChicagoHAI/CaseSumm

CaseSumm: A Large-Scale Dataset for Long-Context Summarization from U.S. Supreme Court Opinions

TL;DR

CaseSumm introduces a large-scale long-context legal summarization dataset consisting of 25.6k U.S. Supreme Court opinions and their official syllabuses (1815–2019), enabling evaluation of long-form legal summaries. It provides a comprehensive, ground-truth–driven benchmark by comparing zero-shot and instruction-tuned LLMs (GPT-4 Turbo and Mistral 7b Instruct with LoRA) against official syllabuses and public controls (Oyez, Westlaw) using automatic metrics and expert human evaluation. The study reveals notable mismatches between automatic metrics and human judgments, with GPT-4 summaries often preferred by humans for clarity and accuracy despite lower automatic scores, and fine-tuned Mistral achieving strong automatic performance but greater factual errors. It also shows that LLM-based evaluation tools like G-Eval do not consistently align better with human judgments than traditional metrics, underscoring the continued need for human evaluation in high-stakes domains. CaseSumm is publicly available on HuggingFace and offers a resource to advance long-context legal summarization research and evaluation methodology.

Abstract

This paper introduces CaseSumm, a novel dataset for long-context summarization in the legal domain that addresses the need for longer and more complex datasets for summarization evaluation. We collect 25.6K U.S. Supreme Court (SCOTUS) opinions and their official summaries, known as "syllabuses." Our dataset is the largest open legal case summarization dataset, and is the first to include summaries of SCOTUS decisions dating back to 1815. We also present a comprehensive evaluation of LLM-generated summaries using both automatic metrics and expert human evaluation, revealing discrepancies between these assessment methods. Our evaluation shows Mistral 7b, a smaller open-source model, outperforms larger models on most automatic metrics and successfully generates syllabus-like summaries. In contrast, human expert annotators indicate that Mistral summaries contain hallucinations. The annotators consistently rank GPT-4 summaries as clearer and exhibiting greater sensitivity and specificity. Further, we find that LLM-based evaluations are not more correlated with human evaluations than traditional automatic metrics. Furthermore, our analysis identifies specific hallucinations in generated summaries, including precedent citation errors and misrepresentations of case facts. These findings demonstrate the limitations of current automatic evaluation methods for legal summarization and highlight the critical role of human evaluation in assessing summary quality, particularly in complex, high-stakes domains. CaseSumm is available at https://huggingface.co/datasets/ChicagoHAI/CaseSumm
Paper Structure (47 sections, 7 figures, 7 tables)

This paper contains 47 sections, 7 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Evaluation for summarization.
  4. NLP and summarization in the Legal Domain.
  5. Datasets in the legal domain.
  6. Dataset
  7. Dataset construction
  8. Descriptive statistics
  9. Experiment Setup
  10. Data and Modeling
  11. Data preprocessing and splits.
  12. Modeling.
  13. Evaluation strategies
  14. "Control group" summaries.
  15. Automatic Evaluation.
  16. ...and 32 more sections

Figures (7)

  • Figure 1: Opinion and syllabus lengths, compression rates by syllabuses, and correlations between opinion and syllabus lengths, 1815-2019. Dashed blue and orange lines give average compression rate and correlation. Lines are smoothed with 5-year moving-average.
  • Figure 2: ROUGE-2 evaluation of model-generated and human summaries, by Chief Justice of SCOTUS when the opinion was written. Markers are means and whiskers are 95% confidence intervals.
  • Figure 3: Human evaluation of model-generated and human summaries. x-axis is a rank, where 1 is best and 5 is worst. For Error, x-axis shows counts of the total number of errors identified by participants for each summary method. See Appendix \ref{['sec:human-eval-dims']} for explanation of each dimension.
  • Figure 4: Lexical Variation. Measures the fraction of words in summary that are not in opinion.
  • Figure 5: Labelstudio Annotation Interface
  • ...and 2 more figures