Towards Comprehensive Stage-wise Benchmarking of Large Language Models in Fact-Checking

TL;DR

FactArena introduces an automated, arena-style framework to benchmark Large Language Models across the full fact-checking pipeline—claim extraction, evidence retrieval, and justification-based verdicts. It employs multi-agent judge panels, consensus guidelines, and arena-driven claim evolution to produce stable, open-form, stage-wise evaluations and robust rankings beyond traditional claim-verification accuracy. Across 16 LLMs from 7 families and 400 complex claims, FactArena demonstrates strong inter-judge reliability, identifies nuanced strengths and weaknesses in stage-wise reasoning, and shows the value of evolving challenging claims for diagnosing factual robustness. The work provides a practical, scalable paradigm for trustworthy auditing of LLMs in safety-critical fact-checking applications and suggests directions for extending evaluation to multimodal claims and uncertainty-aware judgments.

Abstract

Large Language Models (LLMs) are increasingly deployed in real-world fact-checking systems, yet existing evaluations focus predominantly on claim verification and overlook the broader fact-checking workflow, including claim extraction and evidence retrieval. This narrow focus prevents current benchmarks from revealing systematic reasoning failures, factual blind spots, and robustness limitations of modern LLMs. To bridge this gap, we present FactArena, a fully automated arena-style evaluation framework that conducts comprehensive, stage-wise benchmarking of LLMs across the complete fact-checking pipeline. FactArena integrates three key components: (i) an LLM-driven fact-checking process that standardizes claim decomposition, evidence retrieval via tool-augmented interactions, and justification-based verdict prediction; (ii) an arena-styled judgment mechanism guided by consolidated reference guidelines to ensure unbiased and consistent pairwise comparisons across heterogeneous judge agents; and (iii) an arena-driven claim-evolution module that adaptively generates more challenging and semantically controlled claims to probe LLMs' factual robustness beyond fixed seed data. Across 16 state-of-the-art LLMs spanning seven model families, FactArena produces stable and interpretable rankings. Our analyses further reveal significant discrepancies between static claim-verification accuracy and end-to-end fact-checking competence, highlighting the necessity of holistic evaluation. The proposed framework offers a scalable and trustworthy paradigm for diagnosing LLMs' factual reasoning, guiding future model development, and advancing the reliable deployment of LLMs in safety-critical fact-checking applications.

Figures (21)

• Figure 1: The comparison of different focuses between our proposed FactArena and traditional solutions in fact-checking evaluation. Different from the traditional solutions that only focus on claim verification, FactArena aims to scrutinize the full stages of the fact-checking pipeline.
• Figure 2: An overview of our proposed FactArena framework. We automatically conduct a comprehensive benchmarking process for large language models in complete fact-checking stages (e.g., claim extraction, evidence retrieval, and claim verification), distinct from previous traditional audits only focused on the claim verification stage, which is only part of the full fact-checking pipeline.
• Figure 3: An illustration of arena-styled stage-wise judgment and arena-driven claim evolution.
• Figure 4: An illustration of the Elo rankings under FactArena and w/o guideline settings. The order of target mLLMs is the ranking of our main result in Table \ref{['tab:main_results']}.
• Figure 5: Elo ratings of target LLMs before and after claim evolution.