Multimodal Fact-Checking: An Agent-based Approach

TL;DR

This work tackles the challenge of explainable multimodal fact-checking by introducing RW-Post, a high-quality real-world dataset that pairs multimodal claims with original posts, structured reasoning, and explicit evidence. Building on RW-Post, the authors propose AgentFact, an agent-based framework that decomposes verification into five specialized agents (Strategy Planning, Text Evidence Retrieval, Image Retrieval and Analysis, Reasoning, and Explanation Generation) operating in an iterative workflow to improve veracity and explainability. Experiments across RW-Post and other benchmarks demonstrate that AgentFact achieves superior accuracy and interpretability, benefiting from explicit evidence grounding and multi-step reasoning. The dataset and framework together enable more reliable, scalable, and transparent multimodal fact-checking, with practical implications for AI-assisted verification systems. Limitations include computational cost, long-horizon reasoning challenges, and potential data leakage concerns, which the authors propose to address in future work.

Abstract

The rapid spread of multimodal misinformation poses a growing challenge for automated fact-checking systems. Existing approaches, including large vision language models (LVLMs) and deep multimodal fusion methods, often fall short due to limited reasoning and shallow evidence utilization. A key bottleneck is the lack of dedicated datasets that provide complete real-world multimodal misinformation instances accompanied by annotated reasoning processes and verifiable evidence. To address this limitation, we introduce RW-Post, a high-quality and explainable dataset for real-world multimodal fact-checking. RW-Post aligns real-world multimodal claims with their original social media posts, preserving the rich contextual information in which the claims are made. In addition, the dataset includes detailed reasoning and explicitly linked evidence, which are derived from human written fact-checking articles via a large language model assisted extraction pipeline, enabling comprehensive verification and explanation. Building upon RW-Post, we propose AgentFact, an agent-based multimodal fact-checking framework designed to emulate the human verification workflow. AgentFact consists of five specialized agents that collaboratively handle key fact-checking subtasks, including strategy planning, high-quality evidence retrieval, visual analysis, reasoning, and explanation generation. These agents are orchestrated through an iterative workflow that alternates between evidence searching and task-aware evidence filtering and reasoning, facilitating strategic decision-making and systematic evidence analysis. Extensive experimental results demonstrate that the synergy between RW-Post and AgentFact substantially improves both the accuracy and interpretability of multimodal fact-checking.

Figures (9)

• Figure 1: RW-Post Dataset: Use Context (purple highlight) helps LLM determine whether the link (pink highlight) is post or evidence and which rationale it supports.
• Figure 2: Examples of image annotations in the fact-checking dataset. Images are labeled as necessary, relevant, or irrelevant based on whether they provide direct visual evidence, contextual information to claim verification.
• Figure 3: Statistics of RW-Post Dataset
• Figure 4: Proposed Multimodal Fact-checking Agents.
• Figure 5: Proposed Multimodal Fact-checking workflow with Agents.