CroMe: Multimodal Fake News Detection using Cross-Modal Tri-Transformer and Metric Learning

TL;DR

CroMe tackles multimodal fake news detection by jointly modeling text and images through encoders (MAE, BLIP2-OPT, BERT) and a Cross-Modal Tri-Transformer with a Proxy Anchor metric-learning module. It defines a total loss $L_{ ext{total}} = L_{ ext{(\hat{y},y)} } + \beta L(X)$ where $L(X)$ is Proxy Anchor Loss that enforces intra-modality clustering with proxies, margins, and scaling. Empirical results on Weibo, Weibo-21, and Politifact show CroMe achieving 0.974 accuracy on Weibo and 0.917 on Weibo-21, with competitive Politifact performance, illustrating the value of combining intra-modal alignment with cross-modal fusion. The work highlights the practical impact of robust multimodal fake-news detection and points to future directions like data augmentation and cross-dataset validation to improve generalization.

Abstract

Multimodal Fake News Detection has received increasing attention recently. Existing methods rely on independently encoded unimodal data and overlook the advantages of capturing intra-modality relationships and integrating inter-modal similarities using advanced techniques. To address these issues, Cross-Modal Tri-Transformer and Metric Learning for Multimodal Fake News Detection (CroMe) is proposed. CroMe utilizes Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models (BLIP2) as encoders to capture detailed text, image and combined image-text representations. The metric learning module employs a proxy anchor method to capture intra-modality relationships while the feature fusion module uses a Cross-Modal and Tri-Transformer for effective integration. The final fake news detector processes the fused features through a classifier to predict the authenticity of the content. Experiments on datasets show that CroMe excels in multimodal fake news detection.

Figures (6)

• Figure 1: Some fake examples from the Weibo and Weibo-21 datasets include: (a) The image shows intra-modality inconsistency. (b) Both the image and text contain intra-modality inconsistencies. (c) The image and text are unrelated, indicating inter-modality inconsistency. (d) The image does not match the text, reflecting inter-modality conflict.
• Figure 2: Overview of the CroMe architecture. Masked Autoencoder (MAE), BERT, and BLIP2-OPT encode multimodal features. Metric learning extracts intra-modal relationships by representing class data points with proxies, where arrow thickness indicates the gradient scale. CMTTF and fake news detector modules use cross-modal fusion and fake news detection.
• Figure 3: Comparison of two different metric learning methods; (a) Proxy as an anchor, and (b) Data point as an anchor. The thickness of arrows in proxy anchor loss indicates the gradient scale determined by the scaling factor $\alpha$.
• Figure 4: Overview of the CMTTF architecture, integrating Cross-Modal Fusion and Tri-Transformer modules to process and combine information from text, image, and image-text data. The architecture captures cross-modal correlations and fuses them to enhance feature representation.
• Figure 5: Parameter analysis for Weibo, Weibo-21 and Politifact dataset using the heatmap.