Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RU22Fact: Optimizing Evidence for Multilingual Explainable Fact-Checking on Russia-Ukraine Conflict

Yirong Zeng, Xiao Ding, Yi Zhao, Xiangyu Li, Jie Zhang, Chao Yao, Ting Liu, Bing Qin

TL;DR

A method based on a Large Language Model to automatically retrieve and summarize evidence from the Web for explainable fact-checking and an end-to-end explainable fact-checking system to verify claims and generate explanations are proposed.

Abstract

Fact-checking is the task of verifying the factuality of a given claim by examining the available evidence. High-quality evidence plays a vital role in enhancing fact-checking systems and facilitating the generation of explanations that are understandable to humans. However, the provision of both sufficient and relevant evidence for explainable fact-checking systems poses a challenge. To tackle this challenge, we propose a method based on a Large Language Model to automatically retrieve and summarize evidence from the Web. Furthermore, we construct RU22Fact, a novel multilingual explainable fact-checking dataset on the Russia-Ukraine conflict in 2022 of 16K samples, each containing real-world claims, optimized evidence, and referenced explanation. To establish a baseline for our dataset, we also develop an end-to-end explainable fact-checking system to verify claims and generate explanations. Experimental results demonstrate the prospect of optimized evidence in increasing fact-checking performance and also indicate the possibility of further progress in the end-to-end claim verification and explanation generation tasks.

RU22Fact: Optimizing Evidence for Multilingual Explainable Fact-Checking on Russia-Ukraine Conflict

TL;DR

A method based on a Large Language Model to automatically retrieve and summarize evidence from the Web for explainable fact-checking and an end-to-end explainable fact-checking system to verify claims and generate explanations are proposed.

Abstract

Fact-checking is the task of verifying the factuality of a given claim by examining the available evidence. High-quality evidence plays a vital role in enhancing fact-checking systems and facilitating the generation of explanations that are understandable to humans. However, the provision of both sufficient and relevant evidence for explainable fact-checking systems poses a challenge. To tackle this challenge, we propose a method based on a Large Language Model to automatically retrieve and summarize evidence from the Web. Furthermore, we construct RU22Fact, a novel multilingual explainable fact-checking dataset on the Russia-Ukraine conflict in 2022 of 16K samples, each containing real-world claims, optimized evidence, and referenced explanation. To establish a baseline for our dataset, we also develop an end-to-end explainable fact-checking system to verify claims and generate explanations. Experimental results demonstrate the prospect of optimized evidence in increasing fact-checking performance and also indicate the possibility of further progress in the end-to-end claim verification and explanation generation tasks.
Paper Structure (22 sections, 5 equations, 5 figures, 9 tables)

This paper contains 22 sections, 5 equations, 5 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Fact-Checking System
  4. Fact-Checking Dataset
  5. Evidence Analysis
  6. Dataset Construction
  7. Data Collection
  8. Data Processing
  9. Task Definition
  10. Fact-Checking System
  11. Evidence Optimization
  12. Claim Verification
  13. Explanation Generation
  14. Experiment
  15. Claim Verification
  16. ...and 7 more sections

Figures (5)

  • Figure 1: An example of fact verification, based on two different pieces of evidence. (a) a return from a search engine (e.g., Google), and (b) a reply generated by LLMs (e.g., New Bing).
  • Figure 2: Overview of system framework. It consists of an auto evidence optimization module, a claim verification module, and an explanation generation module.
  • Figure 3: An example of information leakage, and the red box indicates label leakage.
  • Figure 4: Schematic representations of strong global coherence, weak global coherence and local coherence. "Neu.", "Ent." and "Con." are abbrs for neutral, entails and contradicts. In each column, the upper part means coherence cannot be satisfied, and the lower part means coherence is satisfied.
  • Figure 5: Examples of strong global coherence, weak global coherence and local coherence.