Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FACTS&EVIDENCE: An Interactive Tool for Transparent Fine-Grained Factual Verification of Machine-Generated Text

Varich Boonsanong, Vidhisha Balachandran, Xiaochuang Han, Shangbin Feng, Lucy Lu Wang, Yulia Tsvetkov

TL;DR

Facts&Evidence introduces an interactive, transparent framework for fine-grained factual verification of AI-generated text by decomposing input into atomic claims, retrieving diverse web evidence for each claim, and computing credibility at the sentence and claim levels with explanations. The system provides a user-driven UI with a credibility visualization and configurable evidence sources, backed by a backend pipeline of atomic claim generation, evidence retrieval, source categorization, and factuality judgement with an open API. Evaluations on the FavaBench dataset show that Facts&Evidence outperforms strong baselines, highlighting the benefits of multi-evidence verification and per-claim explanations for consumer trust. While offering improved transparency and user agency, the work acknowledges latency and dependency on external models, with future directions including efficiency gains and explicit confidence measures.

Abstract

With the widespread consumption of AI-generated content, there has been an increased focus on developing automated tools to verify the factual accuracy of such content. However, prior research and tools developed for fact verification treat it as a binary classification or a linear regression problem. Although this is a useful mechanism as part of automatic guardrails in systems, we argue that such tools lack transparency in the prediction reasoning and diversity in source evidence to provide a trustworthy user experience. We develop Facts&Evidence - an interactive and transparent tool for user-driven verification of complex text. The tool facilitates the intricate decision-making involved in fact-verification, presenting its users a breakdown of complex input texts to visualize the credibility of individual claims along with an explanation of model decisions and attribution to multiple, diverse evidence sources. Facts&Evidence aims to empower consumers of machine-generated text and give them agency to understand, verify, selectively trust and use such text.

FACTS&EVIDENCE: An Interactive Tool for Transparent Fine-Grained Factual Verification of Machine-Generated Text

TL;DR

Facts&Evidence introduces an interactive, transparent framework for fine-grained factual verification of AI-generated text by decomposing input into atomic claims, retrieving diverse web evidence for each claim, and computing credibility at the sentence and claim levels with explanations. The system provides a user-driven UI with a credibility visualization and configurable evidence sources, backed by a backend pipeline of atomic claim generation, evidence retrieval, source categorization, and factuality judgement with an open API. Evaluations on the FavaBench dataset show that Facts&Evidence outperforms strong baselines, highlighting the benefits of multi-evidence verification and per-claim explanations for consumer trust. While offering improved transparency and user agency, the work acknowledges latency and dependency on external models, with future directions including efficiency gains and explicit confidence measures.

Abstract

With the widespread consumption of AI-generated content, there has been an increased focus on developing automated tools to verify the factual accuracy of such content. However, prior research and tools developed for fact verification treat it as a binary classification or a linear regression problem. Although this is a useful mechanism as part of automatic guardrails in systems, we argue that such tools lack transparency in the prediction reasoning and diversity in source evidence to provide a trustworthy user experience. We develop Facts&Evidence - an interactive and transparent tool for user-driven verification of complex text. The tool facilitates the intricate decision-making involved in fact-verification, presenting its users a breakdown of complex input texts to visualize the credibility of individual claims along with an explanation of model decisions and attribution to multiple, diverse evidence sources. Facts&Evidence aims to empower consumers of machine-generated text and give them agency to understand, verify, selectively trust and use such text.

Paper Structure

This paper contains 20 sections, 1 equation, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Facts&Evidence -- User Interface
  3. Landing Page:
  4. Upload Panel:
  5. Readme:
  6. Fact Visualization Page:
  7. Credibility Panel:
  8. Evidence Panel:
  9. Facts&Evidence -- Behind the Scenes
  10. Atomic Claim Generation
  11. Evidence Retrieval
  12. Source categorization
  13. Factuality Judgement
  14. Experiments
  15. Evaluation:
  16. ...and 5 more sections

Figures (4)

  • Figure 1: The pipeline figure of Facts&Evidence. The user input for verification goes through atomic claim and query generation, evidence retrieval, and factuality judgement processes. The system output is aggregated from the judgements over atomic claims, providing users with sources of evidence and an overall credibility score.
  • Figure 2: Facts&Evidence Upload Panel
  • Figure 3: Credibility Panel of Facts&Evidence
  • Figure 5: Binary F1 Results on factual error detection. Facts&Evidence improves error prediction accuracy by $\sim$44 points on average across the two subsets.