ClaimDB: A Fact Verification Benchmark over Large Structured Data

TL;DR

ClaimDB introduces a fact-verification benchmark over large-scale structured data, capturing real-world verification challenges where evidence spans millions of records across multiple tables. The authors design a rigorous construction pipeline, leveraging the BIRD NL-to-SQL corpus, aggressive pre-filtering to enforce compositional reasoning, and GPT-5-based claim generation, followed by a panel of multi-organization LLM judges and NEI sampling to ensure quality. Evaluations across 30 models reveal that none surpass ~83% accuracy, with many below 55%, highlighting the gap in reasoning over large databases and the limitations of abstention for high-stakes analysis. The work emphasizes executable reasoning, tool-based verification, and careful data governance, offering a public benchmark, code, and LLM leaderboard to spur progress toward robust verification in real-world, data-rich settings.

Abstract

Despite substantial progress in fact-verification benchmarks, claims grounded in large-scale structured data remain underexplored. In this work, we introduce ClaimDB, the first fact-verification benchmark where the evidence for claims is derived from compositions of millions of records and multiple tables. ClaimDB consists of 80 unique real-life databases covering a wide range of domains, from governance and healthcare to media, education and the natural sciences. At this scale, verification approaches that rely on "reading" the evidence break down, forcing a timely shift toward reasoning in executable programs. We conduct extensive experiments with 30 state-of-the-art proprietary and open-source (below 70B) LLMs and find that none exceed 83% accuracy, with more than half below 55%. Our analysis also reveals that both closed- and open-source models struggle with abstention -- the ability to admit that there is no evidence to decide -- raising doubts about their reliability in high-stakes data analysis. We release the benchmark, code, and the LLM leaderboard at https://claimdb.github.io .

Figures (17)

• Figure 1: Overview of the ClaimDB construction pipeline. We start from the NL-to-SQL BIRD benchmark (Section \ref{['sec:bird']}), execute each query on its respective real-world database, and filter out low-information pairs using the query AST (Sections \ref{['sec:filtering']}, \ref{['sec:qapairs']}). For each remaining Q/A pair, we prompt gpt-5 to generate claims grounded in the gold answer---with some additional context for NEI claims (Section \ref{['sec:claim_gen']}). We then use a panel of LLM judges from Mistral AI, Microsoft, and xAI to retain only high-quality claims (Section \ref{['sec:judges']}). Finally, we apply embedding-based post-processing to two NEI categories for high-quality sampling (Section \ref{['sec:embeddings']}); this step is omitted for space reasons.
• Figure 2: Claim distribution and taxonomy. We group the 80 databases into 11 high-level domains (introduced in this work), each comprising multiple subdomains. The subdomains are inherited from DBLP:conf/nips/LiHQYLLWQGHZ0LC23 with a few minor modifications.
• Figure 3: Example of claims generated from a single Q/A pair in ClaimDB (California Schools database). In the generation of NEI claims we also give the database schema information along the golden context.
• Figure 4: Performance of the judge panel on the human-annotated test set (75 NEI and 75 C or E). The panel achieves $100\%$ recall on the examples that the human annotators found problematic and behaves conservatively.
• Figure 5: Distribution of smilarity scores between generated claims and their gold context for out-of-schema claims. Higher scores indicate that claims stay "closer" to the underlying data concepts. The two highlighted claim examples ($\clubsuit$, $\bigstar$) are discussed in Section \ref{['sec:embeddings']}.