FaStfact: Faster, Stronger Long-Form Factuality Evaluations in LLMs

TL;DR

FaStfact tackles the inefficiency and unreliability of prior long-form factuality evaluations by introducing chunk-based claim extraction with confidence-based pre-verification, document-level evidence search, and retrieval-augmented verification. It replaces sentence-level extraction and snippet-based evidence with a chunked workflow and full-document retrieval, significantly reducing LLM calls and token costs while improving alignment with human judgments. The authors define a ground-truth-aware F1@K' metric with a symmetrical penalty to avoid verbosity bias and leverage FaStfact-Bench, a 400-sample annotated benchmark, to demonstrate strong, scalable factuality evaluation across diverse domains. The approach yields a practical, faster, and more reliable evaluation framework with a publicly available benchmark and tools for reproducibility and future improvements in LLM factuality assessment.

Abstract

Evaluating the factuality of long-form generations from Large Language Models (LLMs) remains challenging due to efficiency bottlenecks and reliability concerns. Prior efforts attempt this by decomposing text into claims, searching for evidence, and verifying claims, but suffer from critical drawbacks: (1) inefficiency due to overcomplicated pipeline components, and (2) ineffectiveness stemming from inaccurate claim sets and insufficient evidence. To address these limitations, we propose \textbf{FaStfact}, an evaluation framework that achieves the highest alignment with human evaluation and time/token efficiency among existing baselines. FaStfact first employs chunk-level claim extraction integrated with confidence-based pre-verification, significantly reducing the time and token cost while ensuring reliability. For searching and verification, it collects document-level evidence from crawled web-pages and selectively retrieves it during verification. Extensive experiments based on an annotated benchmark \textbf{FaStfact-Bench} demonstrate the reliability of FaStfact in both efficiently and effectively evaluating long-form factuality. Code, benchmark data, and annotation interface tool are available at https://github.com/Yingjia-Wan/FaStfact.

Figures (5)

• Figure 1: Overview of FaStfact. Step 1: An extractor receives a chunk of a long-form generation to extract and pre-verify atomic claims. For each claim, if the pre-verification confidence $C$ falls below a predefined threshold $\theta$, proceed to step 2 and 3; otherwise, skip the following steps and use pre-verification for claim factuality. Step 2: A web search is triggered to crawl document-level evidence for each claim. Step 3: Based on the gathered evidence, a verifier classifies the factual correcteness of each claim. A final factuality score $F_{1}@K'$ (\ref{['subsec: metric']}) is then computed by aggregating the claim-level verifications.
• Figure 2: Breakdown of LLM Inference Calls and Search Queries by Pipeline Phase for different metrics.
• Figure 3: Ablations of altering chunk stride from 1 (= sentence-level window) to MAX (= entire response). The left panel shows the changes in claim extraction reliability, i.e., the discrepancy between FaStfact's number of extracted claims (in solid lines) against ground-truth values (in horizontal dotted lines); the right panel shows the changes in efficiency i.e., token cost.
• Figure 4: Factuality leaderboard on FaStfact-Bench using FaStfact. The dotted line illustrates the average number of claims decomposed by FaStfact.
• Figure 6: Distribution of unverifiable and redundant claims as well as their subcategories, extracted by VeriScore from sampled model generations. Redundant claims include the labels: 'Inter-sentence' and 'Intra-sentence'; Unverifiable claims include the labels: 'Sujective', 'Ambiguous', and 'Tautology'.