Unveiling Modality Bias: Automated Sample-Specific Analysis for Multimodal Misinformation Benchmarks

TL;DR

This work tackles the problem of modality bias in multimodal misinformation benchmarks by introducing an automated sample-specific analysis framework with three complementary views: modality benefit (Shapley-value based), modality flow (saliency-based information flow), and modality causal effect (counterfactual reasoning). An ensemble voting scheme combines the three views to yield robust, sample-level bias labels, validated against human judgments on Fakeddit and MMFakeBench. Key findings show that multi-view ensembles outperform single views, detector choices induce fluctuations, and views agree more on modality-balanced samples than on biased ones, informing dataset cleaning and robustness strategies. The approach scales to real-world and synthetic benchmarks and has potential extensions to video modalities and larger models, offering a practical pathway to diagnose and mitigate modality bias in multimodal misinformation analysis.

Abstract

Numerous multimodal misinformation benchmarks exhibit bias toward specific modalities, allowing detectors to make predictions based solely on one modality. While previous research has quantified bias at the dataset level or manually identified spurious correlations between modalities and labels, these approaches lack meaningful insights at the sample level and struggle to scale to the vast amount of online information. In this paper, we investigate the design for automated recognition of modality bias at the sample level. Specifically, we propose three bias quantification methods based on theories/views of different levels of granularity: 1) a coarse-grained evaluation of modality benefit; 2) a medium-grained quantification of information flow; and 3) a fine-grained causality analysis. To verify the effectiveness, we conduct a human evaluation on two popular benchmarks. Experimental results reveal three interesting findings that provide potential direction toward future research: 1)~Ensembling multiple views is crucial for reliable automated analysis; 2)~Automated analysis is prone to detector-induced fluctuations; and 3)~Different views produce a higher agreement on modality-balanced samples but diverge on biased ones.

Figures (7)

• Figure 1: Effect of different samples on misinformation detection. For biased samples, we can directly infer from preferred modality, like the Left (an unreasonable fat cat image) and Right (the impossibility of resurrection) one.
• Figure 2: Illustration of proposed automated analysis for modality bias in multimodal misinformation benchmarks.
• Figure 3: The Venn diagram of three single-views on Fakeddit (top three) and MMFakeBench (bottom three).
• Figure 4: Accuracy of the view of modality flow with varying threshold $\epsilon$ on Fakeddit and MMFakeBench.
• Figure 5: Examples of core information extraction.