Enhancing Multimodal Misinformation Detection by Replaying the Whole Story from Image Modality Perspective

TL;DR

This work demonstrates that in multimodal misinformation detection, text typically carries more veracity-related information than images. It introduces RETSIMD, which splits text into $K$ segments, generates a sequence of augmented images via a text-to-image generator to replay the story, and optimizes with mutual-information objectives while post-training on large text-image data. A graph-based fusion network then integrates original and generated images using central, temporal, and dependency relationships to improve detection. Across three benchmarks, RETSIMD consistently enhances baseline MMD models and increases the image modality’s contribution, validating the approach and offering a practical path to more robust misinformation moderation.

Abstract

Multimodal Misinformation Detection (MMD) refers to the task of detecting social media posts involving misinformation, where the post often contains text and image modalities. However, by observing the MMD posts, we hold that the text modality may be much more informative than the image modality because the text generally describes the whole event/story of the current post but the image often presents partial scenes only. Our preliminary empirical results indicate that the image modality exactly contributes less to MMD. Upon this idea, we propose a new MMD method named RETSIMD. Specifically, we suppose that each text can be divided into several segments, and each text segment describes a partial scene that can be presented by an image. Accordingly, we split the text into a sequence of segments, and feed these segments into a pre-trained text-to-image generator to augment a sequence of images. We further incorporate two auxiliary objectives concerning text-image and image-label mutual information, and further post-train the generator over an auxiliary text-to-image generation benchmark dataset. Additionally, we propose a graph structure by defining three heuristic relationships between images, and use a graph neural network to generate the fused features. Extensive empirical results validate the effectiveness of RETSIMD.

Figures (6)

• Figure 1: Two real-world MMD post examples. Each text describes the whole story of the current post but the image may only present a partial scene corresponding to the text segment marked in red.
• Figure 2: Preliminary empirical evaluations on accuracy gaps between four ablative variants and the full variant.
• Figure 3: Preliminary empirical evaluations on contribution degrees of four ablative variants.
• Figure 4: Overall framework of RetSimd. We split $\mathbf{x}^t$ into $K$ segments, and use a text-to-image generator to generate corresponding images $\mathbf{x}^g$. Then, we design a multimodal fusion network to integrate these features and predict the veracity label.
• Figure 5: Sensitivity analysis of the segmentation strategies.