Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MUSER: A MUlti-Step Evidence Retrieval Enhancement Framework for Fake News Detection

Hao Liao, Jiaohao Peng, Zhanyi Huang, Wei Zhang, Guanghua Li, Kai Shu, Xing Xie

TL;DR

The paper tackles the challenge of reliable fake-news detection by proposing MUSER, a multi-step evidence retrieval framework that mirrors human verification: summarizing the article, performing iterative retrieval to gather evidence, and reasoning over the evidence to assess truthfulness. It combines a text summarization module, an adaptive multi-step retrieval module with key-evidence selection, and a text reasoning module to produce verified predictions, while providing interpretable, evidence-based justifications. The authors demonstrate that MUSER outperforms state-of-the-art baselines on three real-world multilingual datasets, with about a 3% gain in F1-Macro and F1-Micro, and show through ablation and explainability studies that the multi-step retrieval and summarization components are essential for both performance and interpretability. The approach reduces manual evidence collection, supports early detection without social context, and enhances user trust by attaching transparent supporting evidence to its verdicts.

Abstract

The ease of spreading false information online enables individuals with malicious intent to manipulate public opinion and destabilize social stability. Recently, fake news detection based on evidence retrieval has gained popularity in an effort to identify fake news reliably and reduce its impact. Evidence retrieval-based methods can improve the reliability of fake news detection by computing the textual consistency between the evidence and the claim in the news. In this paper, we propose a framework for fake news detection based on MUlti-Step Evidence Retrieval enhancement (MUSER), which simulates the steps of human beings in the process of reading news, summarizing, consulting materials, and inferring whether the news is true or fake. Our model can explicitly model dependencies among multiple pieces of evidence, and perform multi-step associations for the evidence required for news verification through multi-step retrieval. In addition, our model is able to automatically collect existing evidence through paragraph retrieval and key evidence selection, which can save the tedious process of manual evidence collection. We conducted extensive experiments on real-world datasets in different languages, and the results demonstrate that our proposed model outperforms state-of-the-art baseline methods for detecting fake news by at least 3% in F1-Macro and 4% in F1-Micro. Furthermore, it provides interpretable evidence for end users.

MUSER: A MUlti-Step Evidence Retrieval Enhancement Framework for Fake News Detection

TL;DR

The paper tackles the challenge of reliable fake-news detection by proposing MUSER, a multi-step evidence retrieval framework that mirrors human verification: summarizing the article, performing iterative retrieval to gather evidence, and reasoning over the evidence to assess truthfulness. It combines a text summarization module, an adaptive multi-step retrieval module with key-evidence selection, and a text reasoning module to produce verified predictions, while providing interpretable, evidence-based justifications. The authors demonstrate that MUSER outperforms state-of-the-art baselines on three real-world multilingual datasets, with about a 3% gain in F1-Macro and F1-Micro, and show through ablation and explainability studies that the multi-step retrieval and summarization components are essential for both performance and interpretability. The approach reduces manual evidence collection, supports early detection without social context, and enhances user trust by attaching transparent supporting evidence to its verdicts.

Abstract

The ease of spreading false information online enables individuals with malicious intent to manipulate public opinion and destabilize social stability. Recently, fake news detection based on evidence retrieval has gained popularity in an effort to identify fake news reliably and reduce its impact. Evidence retrieval-based methods can improve the reliability of fake news detection by computing the textual consistency between the evidence and the claim in the news. In this paper, we propose a framework for fake news detection based on MUlti-Step Evidence Retrieval enhancement (MUSER), which simulates the steps of human beings in the process of reading news, summarizing, consulting materials, and inferring whether the news is true or fake. Our model can explicitly model dependencies among multiple pieces of evidence, and perform multi-step associations for the evidence required for news verification through multi-step retrieval. In addition, our model is able to automatically collect existing evidence through paragraph retrieval and key evidence selection, which can save the tedious process of manual evidence collection. We conducted extensive experiments on real-world datasets in different languages, and the results demonstrate that our proposed model outperforms state-of-the-art baseline methods for detecting fake news by at least 3% in F1-Macro and 4% in F1-Micro. Furthermore, it provides interpretable evidence for end users.
Paper Structure (30 sections, 4 equations, 5 figures, 9 tables)

This paper contains 30 sections, 4 equations, 5 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Fake News Detection
  4. Retrieval Enhancement
  5. Natural Language Inference
  6. Problem Statement
  7. The Proposed Model
  8. Text Summarization Module
  9. Text Reasoning Module
  10. EXPERIMENTS
  11. Experimental Setup
  12. Datasets.
  13. Baselines
  14. Implementation Details
  15. Performance Results (RQ1)
  16. ...and 15 more sections

Figures (5)

  • Figure 1: A motivating example of MUSER model. Our model simulates a human evaluating news through three steps: (1) Summarization of the key information, (2) Retrieval and evaluation of relevant evidence: the model assesses the sufficiency and quality of the evidence, determining if additional inquiries are necessary, (3) Conclusion regarding the truthfulness of the news based on the gathered evidence.
  • Figure 2: Our framework unfolds in three steps: (a) Summarization of the initial news text to obtain the key statement $C$, corresponding to the human process of summarizing key information, (b) Evidence finding through multi-step retrieval, corresponding to the human process of querying external relevant information based on the news claim. The retriever sends the first $k$ paragraphs to the evidence selector, which evaluates whether the evidence meets the requirements. The correlation coefficient between $C$ and evidence snippets is represented by $r(c,e)$, and a settable correlation score threshold, $\lambda$, is used to judge the quality of the evidence, and (c) The textual reasoner infers the consistency of evidence and claims, corresponding to the human process of judging news based on evidence.
  • Figure 3: Results of retrieve step comparison study. The term SC (Step Control) means that the key evidence selection function is activated, while WSC (Without Step Control) means that the key evidence selection function is not included.
  • Figure 4: Results of ablation study. MUSER represents the complete model performance, MUSER-RM represents the removal of the multi-step retrieval module and MUSER-RS represents the removal of the text summary module.
  • Figure 5: A verification example generated by MUSER in the Case study. The evidence correlation score $r(c,e)$ obtained by the first step of retrieval is smaller than the threshold $\lambda$ we set. Then proceed to the second step of retrieval to obtain more sufficient evidence.