SafePassage: High-Fidelity Information Extraction with Black Box LLMs

TL;DR

SafePassage tackles the problem of grounded information extraction with black-box LLMs by introducing the safe passage concept and a three-stage pipeline (generation, alignment, scoring) to ensure every extraction is supported by document evidence. It combines an LLM extractor, local sequence alignment, and either an NLI model or an LLM grader to filter out hallucinations, enabling both offline evaluation and online guardrails. Key findings show that a small, task-specific encoder trained with limited human data can outperform an LLM grader while being far cheaper and faster, and that the approach can reduce hallucinations in legal IE by up to 85%. The work demonstrates practical impact for safe deployment of LLMs in professional domains and provides a scalable path to compare LLMs and support systems via SafePassage scoring.

Abstract

Black box large language models (LLMs) make information extraction (IE) easy to configure, but hard to trust. Unlike traditional information extraction pipelines, the information "extracted" is not guaranteed to be grounded in the document. To prevent this, this paper introduces the notion of a "safe passage": context generated by the LLM that is both grounded in the document and consistent with the extracted information. This is operationalized via a three-step pipeline, SafePassage, which consists of: (1) an LLM extractor that generates structured entities and their contexts from a document, (2) a string-based global aligner, and (3) a scoring model. Results show that using these three parts in conjunction reduces hallucinations by up to 85% on information extraction tasks with minimal risk of flagging non-hallucinations. High agreement between the SafePassage pipeline and human judgments of extraction quality mean that the pipeline can be dually used to evaluate LLMs. Surprisingly, results also show that using a transformer encoder fine-tuned on a small number of task-specific examples can outperform an LLM scoring model at flagging unsafe passages. These annotations can be collected in as little as 1-2 hours.

Figures (5)

• Figure 1: An overview of the SafePassage pipeline. A generative language model generates a list of extracted entities and their contexts, which are aligned with the original document, or rejected if they cannot be found in the document. The remaining entities and their aligned contexts are scored by some model (e.g. an NLI model or another LLM), and the low-scoring ones are rejected.
• Figure 2: A sample schema for information extraction with LLMs. Each Entity has a context that gets populated with exact text from the document, for later alignment. Using this approach for information extraction allows you to extract structured values, such as the Date composed of year, month, and day, that can be used downstream. Changing which and how many entities are extracted only requires changing the code for entities and the base prompt.
• Figure 3: Local (Smith–Waterman) alignment of the LLM-provided context to document text. Matches are marked with “|”. In this example, the best local alignment has length $L{=}35$ and $M{=}31$ character matches, so the score is $s{=}M/L{=}0.886$.
• Figure 4: Finetuned sentence transformer models across dataset sizes. At dataset size 0, the base NLI model is being used without finetuning on human judgments. Pretraining with synthetic data from the LLM scorer improves the NLI scorer at all sizes ($\text{NLI}_{\text{synth>human}}$ vs $\text{NLI}_{\text{human}}$). With just 500 labels (roughly 2 hours of annotation effort) both models perform as well as or better than the LLM scorer.
• Figure 5: Average SafePassage scores and average human scores for each of the considered LLMs. Performance trends upward with model size within each group of providers. The best-performing model is the single reasoning model evaluated, o4-mini.