Multimodal Fake News Detection: MFND Dataset and Shallow-Deep Multitask Learning

TL;DR

This work tackles multimodal fake news detection under realistic deepfake threats by introducing MFND, a large-scale dataset with four modality combinations and 11 manipulation types that supports detection and localization. It then proposes SDML, a Shallow-Deep Multitask Learning framework that combines a light, contrastive-alignment–driven shallow path with a two-branch deep-inference path to jointly detect and localize image and text forgery, optimizing with a holistic loss $\\mathcal{L}_{TOTAL}= \\mathcal{L}_{LC}+\\mathcal{L}_{BIC}+\\mathcal{L}_{IMG}+\\mathcal{L}_{TEX}$. Empirical results on MFND and DGM^4 show SDML achieving state-of-the-art performance across multimodal, image, and text tasks, with substantial improvements in localization (IoU). Ablation studies confirm the importance of LPCL, ACMF, and the multi-view extractor, validating that early multimodal alignment combined with deep reasoning improves robustness to realistic manipulations. The work provides a realistic benchmark and a scalable framework for reliable multimodal fake-news detection and localization with practical implications for media credibility and safety.

Abstract

Multimodal news contains a wealth of information and is easily affected by deepfake modeling attacks. To combat the latest image and text generation methods, we present a new Multimodal Fake News Detection dataset (MFND) containing 11 manipulated types, designed to detect and localize highly authentic fake news. Furthermore, we propose a Shallow-Deep Multitask Learning (SDML) model for fake news, which fully uses unimodal and mutual modal features to mine the intrinsic semantics of news. Under shallow inference, we propose the momentum distillation-based light punishment contrastive learning for fine-grained uniform spatial image and text semantic alignment, and an adaptive cross-modal fusion module to enhance mutual modal features. Under deep inference, we design a two-branch framework to augment the image and text unimodal features, respectively merging with mutual modalities features, for four predictions via dedicated detection and localization projections. Experiments on both mainstream and our proposed datasets demonstrate the superiority of the model. Codes and dataset are released at https://github.com/yunan-wang33/sdml.

Figures (8)

• Figure 1: Illustrates of the news from the MFND dataset. (a) Real News of Real Image Real Text, (b) Fake News of Fake Image Real Text, (c) Fake News of Real Image Fake Text, (d) Fake News of Fake Image Fake Text.
• Figure 3: Illustration of the proposed Shallow-Deep Multitask Learning (SDML) method. As for the shallow inference with green lines, we encode two single modalities using different pre-trained Encoders, align the embeddings by contrastive learning, and obtain mutual modality after adaptive fusion for media news binary classification. As for the deep inference with red lines, we enhance features from image and text modalities in a two-branch framework, combined with the fusion feature for detection and localization.
• Figure 4: Ablation on layer numbers of text encoder and Contextual Aggregator.
• Figure 5: Ablation on numbers of Multi-view Extractor.
• Figure 6: Visualization of our predictions (real in green Checkmark while fake in red XSolid). The ground truth and prediction bounding boxes in the images are in yellow and blue, respectively.