Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CaseFacts: A Benchmark for Legal Fact-Checking and Precedent Retrieval

Akshith Reddy Putta, Jacob Devasier, Chengkai Li

TL;DR

CaseFacts presents the first large-scale benchmark for verifying colloquial legal claims against Supreme Court precedents, addressing retrieval across a semantic gap, hard-negative reasoning, and temporal validity via the Overruled class. The authors construct a rigorous synthetic data pipeline using open-weight LLMs to generate and validate claim-evidence pairs from 3,299 SCOTUS cases sourced from Oyez, applying intra- and inter-case checks and a similarity-based filtering strategy to identify overrulings. Evaluation shows that unrestricted web search can hurt performance by surfacing noisy precedents, while fine-tuned semantic-similarity retrievers substantially improve evidence retrieval; however, the overall task remains challenging for state-of-the-art models. The work provides a reusable dataset, a robust pipeline, and baseline results that highlight the need for specialized legal retrieval and verified evidence grounding in high-stakes fact-checking. These contributions lay groundwork for safer, more reliable legal AI systems and encourage future exploration of retrieval-verified verification in dynamic legal domains.

Abstract

Automated Fact-Checking has largely focused on verifying general knowledge against static corpora, overlooking high-stakes domains like law where truth is evolving and technically complex. We introduce CaseFacts, a benchmark for verifying colloquial legal claims against U.S. Supreme Court precedents. Unlike existing resources that map formal texts to formal texts, CaseFacts challenges systems to bridge the semantic gap between layperson assertions and technical jurisprudence while accounting for temporal validity. The dataset consists of 6,294 claims categorized as Supported, Refuted, or Overruled. We construct this benchmark using a multi-stage pipeline that leverages Large Language Models (LLMs) to synthesize claims from expert case summaries, employing a novel semantic similarity heuristic to efficiently identify and verify complex legal overrulings. Experiments with state-of-the-art LLMs reveal that the task remains challenging; notably, augmenting models with unrestricted web search degrades performance compared to closed-book baselines due to the retrieval of noisy, non-authoritative precedents. We release CaseFacts to spur research into legal fact verification systems.

CaseFacts: A Benchmark for Legal Fact-Checking and Precedent Retrieval

TL;DR

CaseFacts presents the first large-scale benchmark for verifying colloquial legal claims against Supreme Court precedents, addressing retrieval across a semantic gap, hard-negative reasoning, and temporal validity via the Overruled class. The authors construct a rigorous synthetic data pipeline using open-weight LLMs to generate and validate claim-evidence pairs from 3,299 SCOTUS cases sourced from Oyez, applying intra- and inter-case checks and a similarity-based filtering strategy to identify overrulings. Evaluation shows that unrestricted web search can hurt performance by surfacing noisy precedents, while fine-tuned semantic-similarity retrievers substantially improve evidence retrieval; however, the overall task remains challenging for state-of-the-art models. The work provides a reusable dataset, a robust pipeline, and baseline results that highlight the need for specialized legal retrieval and verified evidence grounding in high-stakes fact-checking. These contributions lay groundwork for safer, more reliable legal AI systems and encourage future exploration of retrieval-verified verification in dynamic legal domains.

Abstract

Automated Fact-Checking has largely focused on verifying general knowledge against static corpora, overlooking high-stakes domains like law where truth is evolving and technically complex. We introduce CaseFacts, a benchmark for verifying colloquial legal claims against U.S. Supreme Court precedents. Unlike existing resources that map formal texts to formal texts, CaseFacts challenges systems to bridge the semantic gap between layperson assertions and technical jurisprudence while accounting for temporal validity. The dataset consists of 6,294 claims categorized as Supported, Refuted, or Overruled. We construct this benchmark using a multi-stage pipeline that leverages Large Language Models (LLMs) to synthesize claims from expert case summaries, employing a novel semantic similarity heuristic to efficiently identify and verify complex legal overrulings. Experiments with state-of-the-art LLMs reveal that the task remains challenging; notably, augmenting models with unrestricted web search degrades performance compared to closed-book baselines due to the retrieval of noisy, non-authoritative precedents. We release CaseFacts to spur research into legal fact verification systems.
Paper Structure (50 sections, 1 figure, 6 tables)

This paper contains 50 sections, 1 figure, 6 tables.

Table of Contents

  1. Introduction
  2. CaseFacts Benchmark
  3. Task Formulation
  4. Data Source
  5. Dataset Creation Pipeline
  6. Initial Claim Generation
  7. Intra-Case Validation
  8. Factuality Check
  9. Claim Inconsistencies
  10. Intra-Case Inconsistency Resolution
  11. Inter-Case Validation
  12. Inadequacy of Standard NLI
  13. Similarity-Based Filtering
  14. Identifying Overrulings
  15. Identifying Confirmations
  16. ...and 35 more sections

Figures (1)

  • Figure 1: Graph displaying the changes in character length before and after the LLM pass with prompt \ref{['lst:negation_len_fix']}. The length of the supported claims, which negations are generated from, is provided for comparison.