Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VeriTaS: The First Dynamic Benchmark for Multimodal Automated Fact-Checking

Mark Rothermel, Marcus Kornmann, Marcus Rohrbach, Anna Rohrbach

TL;DR

VeriTaS presents the first dynamic, leakage-resistant benchmark for multimodal Automated Fact-Checking by automating a seven-stage pipeline that transforms real-world ClaimReviews into multilingual, media-rich claims with disentangled, uncertainty-aware verdicts. The pipeline spans review discovery, publisher credibility, article scraping, appearance retrieval, claim normalization, verdict standardization, and rectification, producing a 24K final corpus across 54 languages. Human evaluation confirms high agreement between automated and expert judgments (MSE ≤ 0.04; ~97% discretized accuracy), while baseline multimodal LLMs reveal substantial room for improvement, especially after knowledge-cutoff data. VeriTaS demonstrates leakage effects in static benchmarks, underscores the necessity of real-world data for ethical evaluation, and outlines an adaptable, quarterly-update framework to enable meaningful, long-term assessment of multimodal AFC systems.

Abstract

The growing scale of online misinformation urgently demands Automated Fact-Checking (AFC). Existing benchmarks for evaluating AFC systems, however, are largely limited in terms of task scope, modalities, domain, language diversity, realism, or coverage of misinformation types. Critically, they are static, thus subject to data leakage as their claims enter the pretraining corpora of LLMs. As a result, benchmark performance no longer reliably reflects the actual ability to verify claims. We introduce Verified Theses and Statements (VeriTaS), the first dynamic benchmark for multimodal AFC, designed to remain robust under ongoing large-scale pretraining of foundation models. VeriTaS currently comprises 24,000 real-world claims from 108 professional fact-checking organizations across 54 languages, covering textual and audiovisual content. Claims are added quarterly via a fully automated seven-stage pipeline that normalizes claim formulation, retrieves original media, and maps heterogeneous expert verdicts to a novel, standardized, and disentangled scoring scheme with textual justifications. Through human evaluation, we demonstrate that the automated annotations closely match human judgments. We commit to update VeriTaS in the future, establishing a leakage-resistant benchmark, supporting meaningful AFC evaluation in the era of rapidly evolving foundation models. We will make the code and data publicly available.

VeriTaS: The First Dynamic Benchmark for Multimodal Automated Fact-Checking

TL;DR

VeriTaS presents the first dynamic, leakage-resistant benchmark for multimodal Automated Fact-Checking by automating a seven-stage pipeline that transforms real-world ClaimReviews into multilingual, media-rich claims with disentangled, uncertainty-aware verdicts. The pipeline spans review discovery, publisher credibility, article scraping, appearance retrieval, claim normalization, verdict standardization, and rectification, producing a 24K final corpus across 54 languages. Human evaluation confirms high agreement between automated and expert judgments (MSE ≤ 0.04; ~97% discretized accuracy), while baseline multimodal LLMs reveal substantial room for improvement, especially after knowledge-cutoff data. VeriTaS demonstrates leakage effects in static benchmarks, underscores the necessity of real-world data for ethical evaluation, and outlines an adaptable, quarterly-update framework to enable meaningful, long-term assessment of multimodal AFC systems.

Abstract

The growing scale of online misinformation urgently demands Automated Fact-Checking (AFC). Existing benchmarks for evaluating AFC systems, however, are largely limited in terms of task scope, modalities, domain, language diversity, realism, or coverage of misinformation types. Critically, they are static, thus subject to data leakage as their claims enter the pretraining corpora of LLMs. As a result, benchmark performance no longer reliably reflects the actual ability to verify claims. We introduce Verified Theses and Statements (VeriTaS), the first dynamic benchmark for multimodal AFC, designed to remain robust under ongoing large-scale pretraining of foundation models. VeriTaS currently comprises 24,000 real-world claims from 108 professional fact-checking organizations across 54 languages, covering textual and audiovisual content. Claims are added quarterly via a fully automated seven-stage pipeline that normalizes claim formulation, retrieves original media, and maps heterogeneous expert verdicts to a novel, standardized, and disentangled scoring scheme with textual justifications. Through human evaluation, we demonstrate that the automated annotations closely match human judgments. We commit to update VeriTaS in the future, establishing a leakage-resistant benchmark, supporting meaningful AFC evaluation in the era of rapidly evolving foundation models. We will make the code and data publicly available.
Paper Structure (51 sections, 18 figures, 8 tables)

This paper contains 51 sections, 18 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Benchmarks by Task Scope and Domain
  4. Benchmarks by Modality and Annotation
  5. Dynamic Claim Datasets
  6. The VeriTaS Construction Pipeline
  7. Stage 1: Review Discovery
  8. Stage 2: Publisher Identification
  9. Stage 3: Article Scraping
  10. Stage 4: Appearance Retrieval
  11. Stage 5: Claim Normalization
  12. Media Filtering.
  13. Claim Reformulation.
  14. Claim Validation.
  15. Stage 6: Verdict Standardization
  16. ...and 36 more sections

Figures (18)

  • Figure 1: The seven stages of VeriTaS repeated on a quarterly basis.
  • Figure 2: Verdict derivation (Stage 6), assessing properties (1) to (4). Negative decisions in (2) or (3) result in early termination. Definitions are shown on the right.
  • Figure 3: Baseline performance without web search on the longitudinal split using a 200-claim moving average window, single runs. Lower is better. Vertical lines indicate knowledge cutoff dates.
  • Figure 4: Origins of ClaimReviews
  • Figure 5: ClaimReview statistics, as obtained by stages 1 and 2, showing all ClaimReviews exposed by Google and Data Commons retrieved by Dec 31, 2025.
  • ...and 13 more figures