Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Frame-Guided Synthetic Claim Generation for Automatic Fact-Checking Using High-Volume Tabular Data

Jacob Devasier, Akshith Putta, Qing Wang, Alankrit Moses, Chengkai Li

TL;DR

This work proposes a novel, frame-guided methodology where algorithms programmatically select significant data points based on six semantic frames to generate realistic claims in English, Chinese, Spanish, and Hindi, and demonstrates through knowledge-probing experiments that LLMs have not memorized these facts.

Abstract

Automated fact-checking benchmarks have largely ignored the challenge of verifying claims against real-world, high-volume structured data, instead focusing on small, curated tables. We introduce a new large-scale, multilingual dataset to address this critical gap. It contains 78,503 synthetic claims grounded in 434 complex OECD tables, which average over 500K rows each. We propose a novel, frame-guided methodology where algorithms programmatically select significant data points based on six semantic frames to generate realistic claims in English, Chinese, Spanish, and Hindi. Crucially, we demonstrate through knowledge-probing experiments that LLMs have not memorized these facts, forcing systems to perform genuine retrieval and reasoning rather than relying on parameterized knowledge. We provide a baseline SQL-generation system and show that our benchmark is highly challenging. Our analysis identifies evidence retrieval as the primary bottleneck, with models struggling to find the correct data in massive tables. This dataset provides a critical new resource for advancing research on this unsolved, real-world problem.

Frame-Guided Synthetic Claim Generation for Automatic Fact-Checking Using High-Volume Tabular Data

TL;DR

This work proposes a novel, frame-guided methodology where algorithms programmatically select significant data points based on six semantic frames to generate realistic claims in English, Chinese, Spanish, and Hindi, and demonstrates through knowledge-probing experiments that LLMs have not memorized these facts.

Abstract

Automated fact-checking benchmarks have largely ignored the challenge of verifying claims against real-world, high-volume structured data, instead focusing on small, curated tables. We introduce a new large-scale, multilingual dataset to address this critical gap. It contains 78,503 synthetic claims grounded in 434 complex OECD tables, which average over 500K rows each. We propose a novel, frame-guided methodology where algorithms programmatically select significant data points based on six semantic frames to generate realistic claims in English, Chinese, Spanish, and Hindi. Crucially, we demonstrate through knowledge-probing experiments that LLMs have not memorized these facts, forcing systems to perform genuine retrieval and reasoning rather than relying on parameterized knowledge. We provide a baseline SQL-generation system and show that our benchmark is highly challenging. Our analysis identifies evidence retrieval as the primary bottleneck, with models struggling to find the correct data in massive tables. This dataset provides a critical new resource for advancing research on this unsolved, real-world problem.
Paper Structure (33 sections, 1 figure, 7 tables)

This paper contains 33 sections, 1 figure, 7 tables.

Table of Contents

  1. Introduction
  2. Dataset Generation
  3. OECD Data Collection
  4. Claim Type Definition
  5. Data Selection
  6. Data Preprocessing and Filtering
  7. Data structure.
  8. Preprocessing.
  9. Time-window filtering.
  10. Measure combinations.
  11. Claim-Specific Data Extraction
  12. Top-K claims.
  13. Constant-Change claims.
  14. Historical-Extreme claims.
  15. Change-in-Rank claims.
  16. ...and 18 more sections

Figures (1)

  • Figure 1: An example of how our system generates factual claims using specific data from a table. Blue and red colors show the data used to create their corresponding claim. Below each claim is the corresponding semantic frame the claim type is grounded in.