Modality Interactive Mixture-of-Experts for Fake News Detection

TL;DR

This work tackles multimodal fake news detection by explicitly modeling text–image modality interactions through unimodal prediction agreement and semantic alignment. It introduces MIMoE-FND, a hierarchical Mixture-of-Experts framework with an interaction gating module that routes each instance to one of four fusion experts (AM, AA, DM, DA) and employs iMoE-based feature refinement and CLIP-guided semantic assessment. Training combines task loss with unimodal and interaction supervision, augmented by router-Z and balance regularizers, enabling robust and interpretable fusion. Experiments on three real-world English and Chinese benchmarks show state-of-the-art accuracy and illustrate the gating mechanism’s behavior across interaction types, underscoring practical impact for mitigating misinformation.

Abstract

The proliferation of fake news on social media platforms disproportionately impacts vulnerable populations, eroding trust, exacerbating inequality, and amplifying harmful narratives. Detecting fake news in multimodal contexts -- where deceptive content combines text and images -- is particularly challenging due to the nuanced interplay between modalities. Existing multimodal fake news detection methods often emphasize cross-modal consistency but ignore the complex interactions between text and visual elements, which may complement, contradict, or independently influence the predicted veracity of a post. To address these challenges, we present Modality Interactive Mixture-of-Experts for Fake News Detection (MIMoE-FND), a novel hierarchical Mixture-of-Experts framework designed to enhance multimodal fake news detection by explicitly modeling modality interactions through an interaction gating mechanism. Our approach models modality interactions by evaluating two key aspects of modality interactions: unimodal prediction agreement and semantic alignment. The hierarchical structure of MIMoE-FND allows for distinct learning pathways tailored to different fusion scenarios, adapting to the unique characteristics of each modality interaction. By tailoring fusion strategies to diverse modality interaction scenarios, MIMoE-FND provides a more robust and nuanced approach to multimodal fake news detection. We evaluate our approach on three real-world benchmarks spanning two languages, demonstrating its superior performance compared to state-of-the-art methods. By enhancing the accuracy and interpretability of fake news detection, MIMoE-FND offers a promising tool to mitigate the spread of misinformation, with the potential to better safeguard vulnerable communities against its harmful effects.

Figures (7)

• Figure 1: Examples of fake news with possible modality interactions. $\hat{y}_{text}$ and $\hat{y}_{img}$ denote the generated unimodal predictions, where positive class indicates fake news ($y=1$)
• Figure 2: The pipeline of MIMoE-FND contains three phases: 1. Feature Extraction & Refinement: BERT and MAE as unimodal encoders, followed by an iMoE module for feature refinement, 2. Modality Interaction Gating: a modality interaction gating network supervised by unimodal prediction agreement and CLIP-guided semantic alignment, 3. Multimodal Fusion & Detection: four iMoE fusion experts to perform modality interaction gated fusion followed by a final classifier.
• Figure 3: Performance metrics (accuracy, fake news F1 score, real news F1 score) for Weibo-21 dataset with different $\beta$ values and $\lambda$ values. Additional parameter analysis (Appendix \ref{['parameter_analysis_app']}) for Weibo and GossipCop are shown in Figure \ref{['fig:beta_analysis_app']} and Figure \ref{['fig:lambda_analysis_app']}.
• Figure 4: Case study illustrating four instances dispatched by the model into their respective fusion experts: Agreed Misalignment (AM), Agreed Alignment (AA), Disagreed Misalignment (DM), and Disagreed Alignment (DA). Each example shows the model's final prediction ($\hat{y}$), unimodal predictions ($\hat{y}_{\text{text}}$, $\hat{y}_{\text{image}}$), and the dispatch vector for modality interactions (AM, AA, DM, DA). The examples demonstrate the model's capability to effectively address modality interaction-specific challenges.
• Figure 5: Performance metrics (accuracy, fake news F1 score, real news F1 score) for the GossipCop and Weibo datasets with different $\beta$ values. We notice that with a coarse $\beta$ parameter searching between 0 and 1, MIMoE-FND can significantly outperform ablated baseline ($\beta = 0$).

Theorems & Definitions (2)

• definition 1: Unimodal Prediction Divergence
• definition 2: Semantic Alignment