Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Claim Extraction for Fact-Checking: Data, Models, and Automated Metrics

Herbert Ullrich, Tomáš Mlynář, Jan Drchal

TL;DR

The paper tackles automated claim extraction for fact-checking by treating it as a one-to-many abstractive generation problem. It introduces FEVERFact, a dataset of 17K atomic claims drawn from 4.4K contextualized Wikipedia sentences, alongside an automated six-metric evaluation framework that includes Atomicity, Fluency, Decontextualization, Faithfulness, Focus, and Coverage, with $F_{fact}$—the harmonic mean of Focus and Coverage—being the centerpiece. It evaluates multiple generation approaches, including QACG baselines, LLM prompting, and T5-based transfer learning, showing that lightweight, trainable models with diverse-beam decoding can approach human-like claim quality while maintaining scalable evaluation. The work provides a reusable evaluation toolkit and demonstrates strong metric validation against human judgments, while discussing ethical considerations of deploying generative claim creators. Overall, FEVERFact and its evaluation framework offer a scalable, data-driven path for advancing check-worthy claim generation and its assessment in real-world misinformation detection tasks.

Abstract

In this paper, we explore the problem of Claim Extraction using one-to-many text generation methods, comparing LLMs, small summarization models finetuned for the task, and a previous NER-centric baseline QACG. As the current publications on Claim Extraction, Fact Extraction, Claim Generation and Check-worthy Claim Detection are quite scattered in their means and terminology, we compile their common objectives, releasing the FEVERFact dataset, with 17K atomic factual claims extracted from 4K contextualised Wikipedia sentences, adapted from the original FEVER. We compile the known objectives into an Evaluation framework of: Atomicity, Fluency, Decontextualization, Faithfulness checked for each generated claim separately, and Focus and Coverage measured against the full set of predicted claims for a single input. For each metric, we implement a scale using a reduction to an already-explored NLP task. We validate our metrics against human grading of generic claims, to see that the model ranking on $F_{fact}$, our hardest metric, did not change and the evaluation framework approximates human grading very closely in terms of $F_1$ and RMSE.

Claim Extraction for Fact-Checking: Data, Models, and Automated Metrics

TL;DR

The paper tackles automated claim extraction for fact-checking by treating it as a one-to-many abstractive generation problem. It introduces FEVERFact, a dataset of 17K atomic claims drawn from 4.4K contextualized Wikipedia sentences, alongside an automated six-metric evaluation framework that includes Atomicity, Fluency, Decontextualization, Faithfulness, Focus, and Coverage, with —the harmonic mean of Focus and Coverage—being the centerpiece. It evaluates multiple generation approaches, including QACG baselines, LLM prompting, and T5-based transfer learning, showing that lightweight, trainable models with diverse-beam decoding can approach human-like claim quality while maintaining scalable evaluation. The work provides a reusable evaluation toolkit and demonstrates strong metric validation against human judgments, while discussing ethical considerations of deploying generative claim creators. Overall, FEVERFact and its evaluation framework offer a scalable, data-driven path for advancing check-worthy claim generation and its assessment in real-world misinformation detection tasks.

Abstract

In this paper, we explore the problem of Claim Extraction using one-to-many text generation methods, comparing LLMs, small summarization models finetuned for the task, and a previous NER-centric baseline QACG. As the current publications on Claim Extraction, Fact Extraction, Claim Generation and Check-worthy Claim Detection are quite scattered in their means and terminology, we compile their common objectives, releasing the FEVERFact dataset, with 17K atomic factual claims extracted from 4K contextualised Wikipedia sentences, adapted from the original FEVER. We compile the known objectives into an Evaluation framework of: Atomicity, Fluency, Decontextualization, Faithfulness checked for each generated claim separately, and Focus and Coverage measured against the full set of predicted claims for a single input. For each metric, we implement a scale using a reduction to an already-explored NLP task. We validate our metrics against human grading of generic claims, to see that the model ranking on , our hardest metric, did not change and the evaluation framework approximates human grading very closely in terms of and RMSE.

Paper Structure

This paper contains 16 sections, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Main Contributions
  3. FEVERFact Dataset
  4. FEVERFact recall
  5. Claim Extraction Models
  6. Evaluation Framework: Claim Metrics
  7. Reference-free evaluation metrics
  8. Reference-based evaluation metrics
  9. $F_{fact}$-value
  10. Results
  11. Model comparison
  12. Metric validation
  13. Inter-Annotator Agreement
  14. Conclusion
  15. Future works
  16. ...and 1 more sections

Figures (4)

  • Figure 1: An example of claim extraction from the FEVERFact dataset constructed from data published by thorne-vlachos-2021-evidence
  • Figure 2: Example of popup help for the claim quality annotation. This example shows the help for the faithfullness metric and simillar popups were provided also for the other ones.
  • Figure 3: Example of the claim quality annotation page. The annotators were asked to grade the quality of the claimss.
  • Figure 4: Example of the multi-claim annotation page. The annotators were asked to evaluate the coverage and focus of the claims.