MMFakeBench: A Mixed-Source Multimodal Misinformation Detection Benchmark for LVLMs

TL;DR

MMFakeBench addresses the gap in evaluating misinformation by introducing the first mixed-source multimodal benchmark, capturing textual, visual, and cross-modal distortions across 12 sub-types with 11,000 image-text pairs and 3,300 real samples. The paper benchmarks 6 detection methods and 15 LVLMs, revealing poor generalization under mixed-source conditions and a notable gap between open-source models and GPT-4V. To close this gap, the authors propose MMD-Agent, an LVLM-based framework that decomposes detection into hierarchical subtasks with internal reasoning and external knowledge retrieval, significantly boosting performance on MMFakeBench. The work provides a practical, realistic evaluation platform and a baseline framework to spur future research in robust, real-world multimodal misinformation detection.

Abstract

Current multimodal misinformation detection (MMD) methods often assume a single source and type of forgery for each sample, which is insufficient for real-world scenarios where multiple forgery sources coexist. The lack of a benchmark for mixed-source misinformation has hindered progress in this field. To address this, we introduce MMFakeBench, the first comprehensive benchmark for mixed-source MMD. MMFakeBench includes 3 critical sources: textual veracity distortion, visual veracity distortion, and cross-modal consistency distortion, along with 12 sub-categories of misinformation forgery types. We further conduct an extensive evaluation of 6 prevalent detection methods and 15 Large Vision-Language Models (LVLMs) on MMFakeBench under a zero-shot setting. The results indicate that current methods struggle under this challenging and realistic mixed-source MMD setting. Additionally, we propose MMD-Agent, a novel approach to integrate the reasoning, action, and tool-use capabilities of LVLM agents, significantly enhancing accuracy and generalization. We believe this study will catalyze future research into more realistic mixed-source multimodal misinformation and provide a fair evaluation of misinformation detection methods.

Figures (15)

• Figure 1: Top: Previous methods often assume a single misinformation source and conduct single-source detection. Bottom: We collaborate generative models and AI tools to build a mixed-source multimodal misinformation benchmark and achieve mixed-source detection.
• Figure 2: Statistics of the MMFakeBench Benchmark.
• Figure 3: Illustrations of using collaborative generative models and AI tools to generate different sources of misinformation.
• Figure 4: Comparison of standard prompting and proposed MMD-Agent. (a) Three examples of multimodal misinformation from distinct sources. (b) LVLMs with standard prompting methods fail to make correct judgments. (c) MMD-Agent instructs LVLMs to decompose mixed-source detection into smaller subtasks, which are solved by integrating model thoughts and environment observation.
• Figure 5: One of the most harmful examples involves a GPT-generated rumor supported by an AI-generated image, which is challenging for LLaVA-1.6-34B. More examples can be found in the Appendix \ref{['Appendix:error_analysis']}.