Gavel: Agent Meets Checklist for Evaluating LLMs on Long-Context Legal Summarization

TL;DR

This work tackles the challenge of evaluating LLMs on long-context legal summarization, where cases span tens to hundreds of thousands of tokens and often require cross-document integration. It introduces Gavel-Ref, a reference-based framework with three components—checklist extraction, residual facts, and writing-style similarity—and a meta-evaluation validating reliability across multiple LLM backbones. The authors systematically benchmark 12 frontier LLMs on 100 legal cases (32K–512K tokens), revealing that even the strongest models struggle with multi-value and rare checklist items and that case length degrades performance, despite 1M-token context capabilities. To reduce reliance on human references, they propose Gavel-Agent, an autonomous agent scaffold that navigates case documents with six tools to extract checklists directly, achieving substantial token efficiency (up to ~36–59% fewer tokens) with only modest drops in performance compared with end-to-end extraction, and highlighting significant headroom for long-horizon agent-based approaches. Data and code are released to enable replication and further research in long-context legal NLP and automated evaluation.

Abstract

Large language models (LLMs) now support contexts of up to 1M tokens, but their effectiveness on complex long-context tasks remains unclear. In this paper, we study multi-document legal case summarization, where a single case often spans many documents totaling 100K-500K tokens. We introduce Gavel-Ref, a reference-based evaluation framework with multi-value checklist evaluation over 26 items, as well as residual fact and writing-style evaluations. Using Gavel-Ref, we go beyond the single aggregate scores reported in prior work and systematically evaluate 12 frontier LLMs on 100 legal cases ranging from 32K to 512K tokens, primarily from 2025. Our results show that even the strongest model, Gemini 2.5 Pro, achieves only around 50 of $S_{\text{Gavel-Ref}}$, highlighting the difficulty of the task. Models perform well on simple checklist items (e.g., filing date) but struggle on multi-value or rare ones such as settlements and monitor reports. As LLMs continue to improve and may surpass human-written summaries -- making human references less reliable -- we develop Gavel-Agent, an efficient and autonomous agent scaffold that equips LLMs with six tools to navigate and extract checklists directly from case documents. With Qwen3, Gavel-Agent reduces token usage by 36% while resulting in only a 7% drop in $S_{\text{checklist}}$ compared to end-to-end extraction with GPT-4.1.

Figures (28)

• Figure 1: Example of evaluating a Gemini 2.5 Pro summary with Gavel-Ref, which contains: checklist evaluation supporting both string-wise and list-wise comparisons, residual fact evaluation, and writing-style evaluation. An interesting finding is that many modern LLMs tend to omits specific names of people or organizations---in this case, the defendant companies; and in other cases even the U.S. president’s name. Light green indicates matched values.
• Figure 2: Benchmarking results of 12 LLMs on long-context legal summarization with our Gavel-Ref framework across case lengths from 32K to 512K tokens. Models are ordered by $S_\text{Gavel-Ref\xspace}$ on all cases. Gemini 2.5 Pro leads, with all top six positions held by proprietary models.
• Figure 3: Gemini 2.5 Pro performance breakdown: top/bottom 5 checklist items by matching score and most frequently over/under-specified items. Overspecification measured as frequency across all 100 cases; underspecification as frequency among cases where human summary includes that item. Dashed lines are medians: 0.49 matching score, 59% overspecification, 70% underspecification.
• Figure 4: Top-5 LLMs' performance across checklist groups, struggling the most on rare items such as related cases and settlements.
• Figure 5: $S_\text{checklist}$ versus total token usage for different methods extracting from case documents.