Credible, Unreliable or Leaked?: Evidence Verification for Enhanced Automated Fact-checking

TL;DR

The paper tackles the problem of evidence quality in automated fact-checking by introducing CREDULE, a large-scale three-class dataset (Credible, Unreliable, Leaked) and EVVER-Net, a neural verifier that flags leaked and unreliable evidence before it informs claims. By augmenting with domain credibility scores from MBFC, EVVER-Net achieves up to $91.5\%$ accuracy on short texts and $94.4\%$ on long texts, showing robust performance across encoder backbones. The authors also reveal that widely used fact-checking datasets contain substantial leaked or unreliable evidence, underscoring the need for evidence filtering in AFC pipelines. The work offers a practical pathway to more realistic, robust AFC systems and highlights future work on multimodal evidence and broader dataset coverage.

Abstract

Automated fact-checking (AFC) is garnering increasing attention by researchers aiming to help fact-checkers combat the increasing spread of misinformation online. While many existing AFC methods incorporate external information from the Web to help examine the veracity of claims, they often overlook the importance of verifying the source and quality of collected "evidence". One overlooked challenge involves the reliance on "leaked evidence", information gathered directly from fact-checking websites and used to train AFC systems, resulting in an unrealistic setting for early misinformation detection. Similarly, the inclusion of information from unreliable sources can undermine the effectiveness of AFC systems. To address these challenges, we present a comprehensive approach to evidence verification and filtering. We create the "CREDible, Unreliable or LEaked" (CREDULE) dataset, which consists of 91,632 articles classified as Credible, Unreliable and Fact checked (Leaked). Additionally, we introduce the EVidence VERification Network (EVVER-Net), trained on CREDULE to detect leaked and unreliable evidence in both short and long texts. EVVER-Net can be used to filter evidence collected from the Web, thus enhancing the robustness of end-to-end AFC systems. We experiment with various language models and show that EVVER-Net can demonstrate impressive performance of up to 91.5% and 94.4% accuracy, while leveraging domain credibility scores along with short or long texts, respectively. Finally, we assess the evidence provided by widely-used fact-checking datasets including LIAR-PLUS, MOCHEG, FACTIFY, NewsCLIPpings+ and VERITE, some of which exhibit concerning rates of leaked and unreliable evidence.

Figures (3)

• Figure 1: Pipeline of automated fact-checking leveraging the proposed Evidence Verification Network.
• Figure 2: Number of articles per year in CREDULE.
• Figure 3: Inference examples from EVVER-Net applied on the evidence of various datasets. We manually included the domain names from which the evidence originates, as they are not provided by the datasets.