Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Supporting Humans in Evaluating AI Summaries of Legal Depositions

Naghmeh Farzi, Laura Dietz, Dave D. Lewis

TL;DR

This work addresses the risk of factual inaccuracy in AI-generated legal deposition summaries by introducing nugget-based extraction and alignment to support end users. It presents a two-workflow HITL prototype: (1) guided comparison of two summaries to identify nugget-level differences, and (2) guided refinement of a single summary guided by nugget evidence and citation support. The approach combines automatic nugget banks, LLM-based nugget judgments, and citation verification to provide actionable guidance, enabling legal professionals to verify, compare, and improve summaries with reduced cognitive load. The long-term goal is to integrate structured nugget feedback into LLM responses, improving first-draft quality and extending the framework to other high-stakes domains.

Abstract

While large language models (LLMs) are increasingly used to summarize long documents, this trend poses significant challenges in the legal domain, where the factual accuracy of deposition summaries is crucial. Nugget-based methods have been shown to be extremely helpful for the automated evaluation of summarization approaches. In this work, we translate these methods to the user side and explore how nuggets could directly assist end users. Although prior systems have demonstrated the promise of nugget-based evaluation, its potential to support end users remains underexplored. Focusing on the legal domain, we present a prototype that leverages a factual nugget-based approach to support legal professionals in two concrete scenarios: (1) determining which of two summaries is better, and (2) manually improving an automatically generated summary.

Supporting Humans in Evaluating AI Summaries of Legal Depositions

TL;DR

This work addresses the risk of factual inaccuracy in AI-generated legal deposition summaries by introducing nugget-based extraction and alignment to support end users. It presents a two-workflow HITL prototype: (1) guided comparison of two summaries to identify nugget-level differences, and (2) guided refinement of a single summary guided by nugget evidence and citation support. The approach combines automatic nugget banks, LLM-based nugget judgments, and citation verification to provide actionable guidance, enabling legal professionals to verify, compare, and improve summaries with reduced cognitive load. The long-term goal is to integrate structured nugget feedback into LLM responses, improving first-draft quality and extending the framework to other high-stakes domains.

Abstract

While large language models (LLMs) are increasingly used to summarize long documents, this trend poses significant challenges in the legal domain, where the factual accuracy of deposition summaries is crucial. Nugget-based methods have been shown to be extremely helpful for the automated evaluation of summarization approaches. In this work, we translate these methods to the user side and explore how nuggets could directly assist end users. Although prior systems have demonstrated the promise of nugget-based evaluation, its potential to support end users remains underexplored. Focusing on the legal domain, we present a prototype that leverages a factual nugget-based approach to support legal professionals in two concrete scenarios: (1) determining which of two summaries is better, and (2) manually improving an automatically generated summary.
Paper Structure (24 sections, 2 figures)

This paper contains 24 sections, 2 figures.

Table of Contents

  1. Introduction
  2. Contributions.
  3. Related Work
  4. Legal Text Summarization
  5. Hallucination and Factual Inaccuracy
  6. Evaluating Legal Summaries
  7. Human-in-the-Loop (HITL) for Verification
  8. System Description
  9. Automated Analysis
  10. Scenario 1: Guided Summary Comparison
  11. User's problem:
  12. The challenge:
  13. How a system could help:
  14. Example summaries:
  15. Prototype walkthrough:
  16. ...and 9 more sections

Figures (2)

  • Figure 1: Currently, users are left unsupported in assessing which of two summaries is better, relying only on the plain text shown in (a) without any color-coded highlights. (b) Guided Comparison interface (Scenario 1) aligns and highlights shared, unique, and missing nuggets.
  • Figure 2: Middle: AI-generated summary; Left: deposition text with supporting spans highlighted; Right: evaluation results providing guidance for summary refinement.