Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VeriFastScore: Speeding up long-form factuality evaluation

Rishanth Rajendhran, Amir Zadeh, Matthew Sarte, Chuan Li, Mohit Iyyer

TL;DR

VeriFastScore tackles the latency of long-form factuality evaluation by replacing the multi-stage, per-claim verification pipeline with a single-pass model fine-tuned on synthetic data derived from VeriScore. It jointly extracts verifiable claims from a long-form response and verifies them against evidence collected from Google Search, handling ~4K tokens of evidence and ~23 claims per response. The approach achieves a high alignment with the original VeriScore (Pearson r ≈ 0.80 at the example level and ≈0.94 system-wide) while delivering a 6.6x speedup (9.9x excluding evidence retrieval) and substantial cost reductions relative to VeriScore. The authors release VeriFastScore and the synthetic datasets to support scalable factuality evaluation and alignment research.

Abstract

Metrics like FactScore and VeriScore that evaluate long-form factuality operate by decomposing an input response into atomic claims and then individually verifying each claim. While effective and interpretable, these methods incur numerous LLM calls and can take upwards of 100 seconds to evaluate a single response, limiting their practicality in large-scale evaluation and training scenarios. To address this, we propose VeriFastScore, which leverages synthetic data to fine-tune Llama3.1 8B for simultaneously extracting and verifying all verifiable claims within a given text based on evidence from Google Search. We show that this task cannot be solved via few-shot prompting with closed LLMs due to its complexity: the model receives ~4K tokens of evidence on average and needs to concurrently decompose claims, judge their verifiability, and verify them against noisy evidence. However, our fine-tuned VeriFastScore model demonstrates strong correlation with the original VeriScore pipeline at both the example level (r=0.80) and system level (r=0.94) while achieving an overall speedup of 6.6x (9.9x excluding evidence retrieval) over VeriScore. To facilitate future factuality research, we publicly release our VeriFastScore model and synthetic datasets.

VeriFastScore: Speeding up long-form factuality evaluation

TL;DR

VeriFastScore tackles the latency of long-form factuality evaluation by replacing the multi-stage, per-claim verification pipeline with a single-pass model fine-tuned on synthetic data derived from VeriScore. It jointly extracts verifiable claims from a long-form response and verifies them against evidence collected from Google Search, handling ~4K tokens of evidence and ~23 claims per response. The approach achieves a high alignment with the original VeriScore (Pearson r ≈ 0.80 at the example level and ≈0.94 system-wide) while delivering a 6.6x speedup (9.9x excluding evidence retrieval) and substantial cost reductions relative to VeriScore. The authors release VeriFastScore and the synthetic datasets to support scalable factuality evaluation and alignment research.

Abstract

Metrics like FactScore and VeriScore that evaluate long-form factuality operate by decomposing an input response into atomic claims and then individually verifying each claim. While effective and interpretable, these methods incur numerous LLM calls and can take upwards of 100 seconds to evaluate a single response, limiting their practicality in large-scale evaluation and training scenarios. To address this, we propose VeriFastScore, which leverages synthetic data to fine-tune Llama3.1 8B for simultaneously extracting and verifying all verifiable claims within a given text based on evidence from Google Search. We show that this task cannot be solved via few-shot prompting with closed LLMs due to its complexity: the model receives ~4K tokens of evidence on average and needs to concurrently decompose claims, judge their verifiability, and verify them against noisy evidence. However, our fine-tuned VeriFastScore model demonstrates strong correlation with the original VeriScore pipeline at both the example level (r=0.80) and system level (r=0.94) while achieving an overall speedup of 6.6x (9.9x excluding evidence retrieval) over VeriScore. To facilitate future factuality research, we publicly release our VeriFastScore model and synthetic datasets.

Paper Structure

This paper contains 56 sections, 4 equations, 2 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Building a faster factuality metric
  3. The "decompose and verify" pipeline for factuality evaluation
  4. VeriScore, our starting point:
  5. Issues with VeriScore
  6. VeriFastScore's Integrated Approach
  7. Evidence Retrieval Without Claim Decomposition:
  8. Simultaneous Claim Decomposition and Verification:
  9. Score calculation:
  10. Generating Synthetic Training Data for VeriFastScore
  11. Data Sources and Motivation:
  12. Prompt Selection and Dataset Choice:
  13. Data Collection Pipeline:
  14. Discrepancy between evidence used during training vs. test-time:
  15. Evidence Overlap and Label Robustness:
  16. ...and 41 more sections

Figures (2)

  • Figure 1: (left) The VeriScore pipeline veriscore involves a separate call to a trained model (or LLM API) to decompose every sentence of the long-form response. More model calls are issued for each decomposed claim to verify them against claim-level evidence retrieved from Google Search. In contrast, our VeriFastScore method retrieves sentence-level evidence from Google Search, combines it together with the response into a single long-context prompt, and outputs a list of claims and labels with just a single call to a trained model.
  • Figure 2: Model rankings produced by VeriScore and VeriFastScore using a random subset of 100 prompts from the test split of Tulu3-Personas data. The rankings produced by both models are fairly consistent with each other despite some differences in order. There is a clear separation between open-weight and closed-weight models.