Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Unified Multimodal Misinformation Detection in Social Media: A Benchmark Dataset and Baseline

Haiyang Li, Yaxiong Wang, Shengeng Tang, Lianwei Wu, Lechao Cheng, Zhun Zhong

TL;DR

This work tackles the dual challenge of detecting both human-crafted and AI-generated multimodal misinformation on social media. It introduces OmniFake, a large-scale benchmark with Real, Human-crafted, and AI-synthesized content to enable unified detection, and UMFDet, a detector built on a Multimodal LLM backbone augmented with a Category-aware Mixture-of-Experts and an Attribution Chain-of-Thought. The method demonstrates strong cross-type performance, outperforming specialized baselines and exhibiting good zero-shot transfer; ablations confirm the contributions of the MoE, deeper MoE, and CoT guidance. The results suggest a practical path toward robust, real-world multimodal deception detection with interpretable reasoning traces.

Abstract

In recent years, detecting fake multimodal content on social media has drawn increasing attention. Two major forms of deception dominate: human-crafted misinformation (e.g., rumors and misleading posts) and AI-generated content produced by image synthesis models or vision-language models (VLMs). Although both share deceptive intent, they are typically studied in isolation. NLP research focuses on human-written misinformation, while the CV community targets AI-generated artifacts. As a result, existing models are often specialized for only one type of fake content. In real-world scenarios, however, the type of a multimodal post is usually unknown, limiting the effectiveness of such specialized systems. To bridge this gap, we construct the Omnibus Dataset for Multimodal News Deception (OmniFake), a comprehensive benchmark of 127K samples that integrates human-curated misinformation from existing resources with newly synthesized AI-generated examples. Based on this dataset, we propose Unified Multimodal Fake Content Detection (UMFDet), a framework designed to handle both forms of deception. UMFDet leverages a VLM backbone augmented with a Category-aware Mixture-of-Experts (MoE) Adapter to capture category-specific cues, and an attribution chain-of-thought mechanism that provides implicit reasoning guidance for locating salient deceptive signals. Extensive experiments demonstrate that UMFDet achieves robust and consistent performance across both misinformation types, outperforming specialized baselines and offering a practical solution for real-world multimodal deception detection.

Towards Unified Multimodal Misinformation Detection in Social Media: A Benchmark Dataset and Baseline

TL;DR

This work tackles the dual challenge of detecting both human-crafted and AI-generated multimodal misinformation on social media. It introduces OmniFake, a large-scale benchmark with Real, Human-crafted, and AI-synthesized content to enable unified detection, and UMFDet, a detector built on a Multimodal LLM backbone augmented with a Category-aware Mixture-of-Experts and an Attribution Chain-of-Thought. The method demonstrates strong cross-type performance, outperforming specialized baselines and exhibiting good zero-shot transfer; ablations confirm the contributions of the MoE, deeper MoE, and CoT guidance. The results suggest a practical path toward robust, real-world multimodal deception detection with interpretable reasoning traces.

Abstract

In recent years, detecting fake multimodal content on social media has drawn increasing attention. Two major forms of deception dominate: human-crafted misinformation (e.g., rumors and misleading posts) and AI-generated content produced by image synthesis models or vision-language models (VLMs). Although both share deceptive intent, they are typically studied in isolation. NLP research focuses on human-written misinformation, while the CV community targets AI-generated artifacts. As a result, existing models are often specialized for only one type of fake content. In real-world scenarios, however, the type of a multimodal post is usually unknown, limiting the effectiveness of such specialized systems. To bridge this gap, we construct the Omnibus Dataset for Multimodal News Deception (OmniFake), a comprehensive benchmark of 127K samples that integrates human-curated misinformation from existing resources with newly synthesized AI-generated examples. Based on this dataset, we propose Unified Multimodal Fake Content Detection (UMFDet), a framework designed to handle both forms of deception. UMFDet leverages a VLM backbone augmented with a Category-aware Mixture-of-Experts (MoE) Adapter to capture category-specific cues, and an attribution chain-of-thought mechanism that provides implicit reasoning guidance for locating salient deceptive signals. Extensive experiments demonstrate that UMFDet achieves robust and consistent performance across both misinformation types, outperforming specialized baselines and offering a practical solution for real-world multimodal deception detection.

Paper Structure

This paper contains 16 sections, 8 equations, 4 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Fake News Detection
  4. Multi-Modal Large Language Models
  5. Omnibus Dataset for Multimodal News Deception Detection
  6. Synthetic Multimodal News
  7. Dataset Statistics
  8. UMFDet Model
  9. Architecture
  10. Instruction Design and Embedding
  11. Category-aware Mixture-of-Expert
  12. Attribution Chain-of-Though Guidance
  13. Experiment
  14. Quantitative Results
  15. Ablation Study
  16. ...and 1 more sections

Figures (4)

  • Figure 1: Comparison of our proposed task, i.e., Ternary multimodal fake news datasets (right) vs. Traditional Binary multimodal fake news datasets (left).Existing datasets → Binary (Real vs. Fake), limited to single-modality forgery vs. Ours → Ternary (Real, Rumor, Manipulated), covers diverse image and text manipulations
  • Figure 2: Dataset pipeline. We construct image--text pairs and expand them via two tracks: (i) image-side manipulations (full-image generation, inpainting, face swap, attribute edits, style transfer) and (ii) LLM-driven text generation/rewriting with controlled cross-modal semantics (aligned or subtly contradictory). Samples are labeled as Real, Human-crafted, or AI-synthesized, enabling unified multimodal detection.
  • Figure 3: Our model is explainable while classifying: it verifies news authenticity (a), flags deceptive content (b), detects and explains image manipulation (c), and identifies text tampering (d).
  • Figure 4: Visualization of expert routing distribution.