What do people want to fact-check?
Bijean Ghafouri, Dorsaf Sallami, Luca Luceri, Taylor Lynn Curtis, Jean-Francois Godbout, Emilio Ferrara, Reihaneh Rabbany
TL;DR
The paper investigates what ordinary people seek to fact-check using an open-ended AI verification system, filling a gap in misinformation research that has focused on supply. By analyzing ~2,500 claims from 457 participants and labeling each along five semantic dimensions, the study reveals that while topics vary widely, users exhibit a narrow epistemic repertoire, mostly submitting simple, present-day, descriptive claims. About 21–25% of queries concern unverifiable or normative content, highlighting a mismatch between what fact-checking tools can adjudicate and the uncertainty people experience. A direct comparison with the FEVER benchmark shows systematic structural differences across domains, epistemic forms, verifiability, targets, and temporal orientation, suggesting that current benchmarks may not reflect real-world verification demands. The findings advocate for demand-driven evaluation and the design of next-generation verification tools capable of signaling when claims fall outside empirical resolution, along with benchmarks that better capture public verification needs.
Abstract
Research on misinformation has focused almost exclusively on supply, asking what falsehoods circulate, who produces them, and whether corrections work. A basic demand-side question remains unanswered. When ordinary people can fact-check anything they want, what do they actually ask about? We provide the first large-scale evidence on this question by analyzing close to 2{,}500 statements submitted by 457 participants to an open-ended AI fact-checking system. Each claim is classified along five semantic dimensions (domain, epistemic form, verifiability, target entity, and temporal reference), producing a behavioral map of public verification demand. Three findings stand out. First, users range widely across topics but default to a narrow epistemic repertoire, overwhelmingly submitting simple descriptive claims about present-day observables. Second, roughly one in four requests concerns statements that cannot be empirically resolved, including moral judgments, speculative predictions, and subjective evaluations, revealing a systematic mismatch between what users seek from fact-checking tools and what such tools can deliver. Third, comparison with the FEVER benchmark dataset exposes sharp structural divergences across all five dimensions, indicating that standard evaluation corpora encode a synthetic claim environment that does not resemble real-world verification needs. These results reframe fact-checking as a demand-driven problem and identify where current AI systems and benchmarks are misaligned with the uncertainty people actually experience.