QuanTemp: A real-world open-domain benchmark for fact-checking numerical claims

TL;DR

QuanTemp introduces the first large-scale, real-world open-domain dataset focused on numerical claim verification, comprising $15{,}514$ claims and an evidence corpus of $423{,}320$ snippets across diverse domains. The authors propose a baseline fact-checking pipeline that combines claim decomposition (ClaimDecomp and Program-FC), BM25 evidence retrieval, and NLI models, and they perform extensive ablations with numerically focused models (e.g., NumT5, FinQA-Roberta-Large). Key findings show that claim decomposition and numeric-understanding NLI models substantially improve verification, especially for complex categories like comparison and interval claims, and that smaller, numerically trained models can outperform larger zero-shot systems. QuanTemp is demonstrated to be a challenging yet practical benchmark with broad applicability, and the dataset along with code is publicly released to spur advances in numerical claim verification.

Abstract

Automated fact checking has gained immense interest to tackle the growing misinformation in the digital era. Existing systems primarily focus on synthetic claims on Wikipedia, and noteworthy progress has also been made on real-world claims. In this work, we release QuanTemp, a diverse, multi-domain dataset focused exclusively on numerical claims, encompassing temporal, statistical and diverse aspects with fine-grained metadata and an evidence collection without leakage. This addresses the challenge of verifying real-world numerical claims, which are complex and often lack precise information, not addressed by existing works that mainly focus on synthetic claims. We evaluate and quantify the limitations of existing solutions for the task of verifying numerical claims. We also evaluate claim decomposition based methods, numerical understanding based models and our best baselines achieves a macro-F1 of 58.32. This demonstrates that QuanTemp serves as a challenging evaluation set for numerical claim verification.

Figures (4)

• Figure 1: Example claim from QuanTemp
• Figure 2: QuanTemp Construction Pipeline
• Figure 2: Top fact-checking domains
• Figure 3: Example of identification of quantitative segments from the claim. NNS is noun plural form, NP-TMP is temporal noun phrase