Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Detecting Manipulated Contents Using Knowledge-Grounded Inference

Mark Huasong Meng, Ruizhe Wang, Meng Xu, Chuan Yan, Guangdong Bai

TL;DR

This work tackles zero-day manipulated content detection by combining real-time information retrieval with knowledge-grounded inference. It introduces Manicod, a two-phase system that autonomously sources internet context and uses a large language model to infer veracity and generate explanations, mitigating the limits of training-time knowledge. A dedicated dataset of 4,270 manipulated headlines derived from 2,500 real headlines is proposed to evaluate zero-day scenarios, with open-source release planned. Empirical results show a strong overall F1 score of 0.856 and substantial improvements over baselines on both binary and multi-class fact-checking tasks, underscoring the practical potential for robust online misinformation defense.

Abstract

The detection of manipulated content, a prevalent form of fake news, has been widely studied in recent years. While existing solutions have been proven effective in fact-checking and analyzing fake news based on historical events, the reliance on either intrinsic knowledge obtained during training or manually curated context hinders them from tackling zero-day manipulated content, which can only be recognized with real-time contextual information. In this work, we propose Manicod, a tool designed for detecting zero-day manipulated content. Manicod first sources contextual information about the input claim from mainstream search engines, and subsequently vectorizes the context for the large language model (LLM) through retrieval-augmented generation (RAG). The LLM-based inference can produce a "truthful" or "manipulated" decision and offer a textual explanation for the decision. To validate the effectiveness of Manicod, we also propose a dataset comprising 4270 pieces of manipulated fake news derived from 2500 recent real-world news headlines. Manicod achieves an overall F1 score of 0.856 on this dataset and outperforms existing methods by up to 1.9x in F1 score on their benchmarks on fact-checking and claim verification.

Detecting Manipulated Contents Using Knowledge-Grounded Inference

TL;DR

This work tackles zero-day manipulated content detection by combining real-time information retrieval with knowledge-grounded inference. It introduces Manicod, a two-phase system that autonomously sources internet context and uses a large language model to infer veracity and generate explanations, mitigating the limits of training-time knowledge. A dedicated dataset of 4,270 manipulated headlines derived from 2,500 real headlines is proposed to evaluate zero-day scenarios, with open-source release planned. Empirical results show a strong overall F1 score of 0.856 and substantial improvements over baselines on both binary and multi-class fact-checking tasks, underscoring the practical potential for robust online misinformation defense.

Abstract

The detection of manipulated content, a prevalent form of fake news, has been widely studied in recent years. While existing solutions have been proven effective in fact-checking and analyzing fake news based on historical events, the reliance on either intrinsic knowledge obtained during training or manually curated context hinders them from tackling zero-day manipulated content, which can only be recognized with real-time contextual information. In this work, we propose Manicod, a tool designed for detecting zero-day manipulated content. Manicod first sources contextual information about the input claim from mainstream search engines, and subsequently vectorizes the context for the large language model (LLM) through retrieval-augmented generation (RAG). The LLM-based inference can produce a "truthful" or "manipulated" decision and offer a textual explanation for the decision. To validate the effectiveness of Manicod, we also propose a dataset comprising 4270 pieces of manipulated fake news derived from 2500 recent real-world news headlines. Manicod achieves an overall F1 score of 0.856 on this dataset and outperforms existing methods by up to 1.9x in F1 score on their benchmarks on fact-checking and claim verification.
Paper Structure (37 sections, 1 equation, 13 figures, 4 tables)

This paper contains 37 sections, 1 equation, 13 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Summary
  3. Background
  4. Disinformation and Misinformation
  5. Problem Definition
  6. Related Work
  7. Textual Veracity Assessment by Conventional ML Models
  8. Disinformation Detection with LLMs
  9. Fairness Evaluations
  10. Our Approach
  11. Online Knowledge Retrieval
  12. Knowledge-Grounded Inference
  13. A Dataset of Manipulated Content
  14. Evaluation
  15. Experiment Settings
  16. ...and 22 more sections

Figures (13)

  • Figure 1: Two examples on how detects and analyzes simulated zero-day manipulated content as of 2024 July 29, with manipulated content in the user queries underlined in red, and key explanations underlined in black.
  • Figure 2: An overview of our disinformation detection framework
  • Figure 3: The prompt template used in with task description, key inference rules, and output instructions highlighted in red, blue, and green, respectively.
  • Figure 4: An overview of our dataset creation
  • Figure 5: Distribution of the 2,500 news collected by regions (Left) and by news providers (Right)
  • ...and 8 more figures