Mitigating GenAI-powered Evidence Pollution for Out-of-Context Multimodal Misinformation Detection

TL;DR

This work investigates how GenAI-generated evidence pollution degrades out-of-context multimodal misinformation detectors in open-domain settings. It demonstrates a significant robustness gap when evidence is polluted and introduces two plug-and-play defenses: cross-modal evidence reranking and cross-modal claim-evidence reasoning. Across NewsCLIPpings and VERITE benchmarks, these strategies markedly improve detection accuracy and resilience against multimodal pollution for detectors like SNIFFER, CCN, and RED-DOT. The results highlight practical approaches to strengthen misinformation detection in the GenAI era, where web-sourced evidence may be contaminated or manipulated.

Abstract

While large generative artificial intelligence (GenAI) models have achieved significant success, they also raise growing concerns about online information security due to their potential misuse for generating deceptive content. Out-of-context (OOC) multimodal misinformation detection, which often retrieves Web evidence to identify the repurposing of images in false contexts, faces the issue of reasoning over GenAI-polluted evidence to derive accurate predictions. Existing works simulate GenAI-powered pollution at the claim level with stylistic rewriting to conceal linguistic cues, and ignore evidence-level pollution for such information-seeking applications. In this work, we investigate how polluted evidence affects the performance of existing OOC detectors, revealing a performance degradation of more than 9 percentage points. We propose two strategies, cross-modal evidence reranking and cross-modal claim-evidence reasoning, to address the challenges posed by polluted evidence. Extensive experiments on two benchmark datasets show that these strategies can effectively enhance the robustness of existing out-of-context detectors amidst polluted evidence.

Figures (11)

• Figure 1: Example of how misinformation detectors are misled by claim-level versus evidence-level pollution posed by GenAI. Faces of individuals are obscured to reduce privacy risks and mitigate the effects of misinformation exposure.
• Figure 2: An illustrated example of claim-conditioned generated evidence, accompanied by clean evidence retrieved from the Web.
• Figure 3: Overview of a typical OOC detection framework.
• Figure 4: OOC misinformation detection framework in the presence of polluted evidence with proposed cross-modal reranking and cross-modal claim-evidence reasoning strategies.
• Figure 5: SNIFFER's performance across varying proportion of polluted evidence and GenAI models on NewsCLIPpings.