DIVER: Dynamic Iterative Visual Evidence Reasoning for Multimodal Fake News Detection

TL;DR

DIVER tackles multimodal fake news detection by introducing a dynamic, evidence-driven reasoning pipeline that prioritizes textual verification and only engaging visual forensics when cross-modal misalignment is detected. It combines linguistic extraction of atomic claims, intra-modal consistency checks, and an inter-modal gating mechanism powered by CLIP to decide when to perform targeted visual forensics (OCR, dense captions), all fused through an uncertainty-aware mechanism. The approach achieves state-of-the-art results on Weibo, Weibo21, and GossipCop with superior inference efficiency, thanks to selective reasoning and robust error handling via a self-correction loop. This dynamic, alignment-guided framework reduces visual hallucination risk and offers interpretable, evidence-backed predictions suitable for real-world misinformation detection.

Abstract

Multimodal fake news detection is crucial for mitigating adversarial misinformation. Existing methods, relying on static fusion or LLMs, face computational redundancy and hallucination risks due to weak visual foundations. To address this, we propose DIVER (Dynamic Iterative Visual Evidence Reasoning), a framework grounded in a progressive, evidence-driven reasoning paradigm. DIVER first establishes a strong text-based baseline through language analysis, leveraging intra-modal consistency to filter unreliable or hallucinated claims. Only when textual evidence is insufficient does the framework introduce visual information, where inter-modal alignment verification adaptively determines whether deeper visual inspection is necessary. For samples exhibiting significant cross-modal semantic discrepancies, DIVER selectively invokes fine-grained visual tools (e.g., OCR and dense captioning) to extract task-relevant evidence, which is iteratively aggregated via uncertainty-aware fusion to refine multimodal reasoning. Experiments on Weibo, Weibo21, and GossipCop demonstrate that DIVER outperforms state-of-the-art baselines by an average of 2.72\%, while optimizing inference efficiency with a reduced latency of 4.12 s.

Figures (8)

• Figure 1: Two primary limitations of existing multi-modal methods: (a) noise induced factual ambiguity, and (b) visual hallucination due to weak grounding.
• Figure 2: Overview of the DIVER framework featuring two explicit feedback loops. The pipeline initiates with linguistic investigation to verify intra-modal consistency. Subsequently, a cross-modal alignment gate dynamically regulates reasoning depth: samples exhibiting high semantic consensus are resolved immediately, while disjoint cases trigger evidence-driven visual forensics, selectively activating visual analysis capabilities (e.g., OCR) to aggregate evidence for claim refinement.
• Figure 3: Analysis of hyperparameter sensitivity. This figure shows the impact of four different hyperparameters on the model's F1-real score across three datasets.
• Figure 4: T-SNE visualization of test set features. Same color dots indicate the same label.
• Figure 5: Analysis of hyperparameter sensitivity. This figure illustrates the impact of four different hyperparameters on the model's F1-Fake score across three datasets.