GAMED: Knowledge Adaptive Multi-Experts Decoupling for Multimodal Fake News Detection

TL;DR

GAMED tackles multimodal fake news detection by decoupling textual and visual information and refining features through a mixture of experts, guided by semantic knowledge. It introduces AdaIN-based adaptive distribution and a veto voting mechanism to enhance transparency and robustness, with external knowledge graphs enriching reasoning. On Fakeddit and Yang, GAMED surpasses state-of-the-art baselines in accuracy and other metrics, and ablation studies confirm the value of each component. The framework offers a scalable, interpretable approach that can extend to additional modalities, advancing practical multimodal misinformation detection.

Abstract

Multimodal fake news detection often involves modelling heterogeneous data sources, such as vision and language. Existing detection methods typically rely on fusion effectiveness and cross-modal consistency to model the content, complicating understanding how each modality affects prediction accuracy. Additionally, these methods are primarily based on static feature modelling, making it difficult to adapt to the dynamic changes and relationships between different data modalities. This paper develops a significantly novel approach, GAMED, for multimodal modelling, which focuses on generating distinctive and discriminative features through modal decoupling to enhance cross-modal synergies, thereby optimizing overall performance in the detection process. GAMED leverages multiple parallel expert networks to refine features and pre-embed semantic knowledge to improve the experts' ability in information selection and viewpoint sharing. Subsequently, the feature distribution of each modality is adaptively adjusted based on the respective experts' opinions. GAMED also introduces a novel classification technique to dynamically manage contributions from different modalities, while improving the explainability of decisions. Experimental results on the Fakeddit and Yang datasets demonstrate that GAMED performs better than recently developed state-of-the-art models. The source code can be accessed at https://github.com/slz0925/GAMED.

Figures (6)

• Figure 1: Starting from raw data, the GAMED's modality-specific pipeline performs feature extraction and progressive refinement. The knowledge enhancement mechanism provides an external background to the architecture. During the expert review stage, features are selected and coarse predictions are made. The AdaIN component then adaptively adjusts the feature distribution. The decisive voting stage orchestrates the final classification.
• Figure 2: Left: The configuration of MMoE-Pro and the flow of processing representations. Right: The pipeline of four modules from coarse prediction to adaptive feature distribution adjustment to obtain enhanced representations.
• Figure 3: Our novel veto model.
• Figure 4: Heatmaps of cosine similarity on Fakeddit and Yang. Each heatmap cell shows the pairwise cosine similarity between the 64-dimensional representation from the coarse predictor of four modules and the whole model.
• Figure 5: The comparison of accuracy (first row) and loss (second row) illustrates the learning curves of GAMED and its four modules during training. The training set on the left column and right column is Fakeddit and Yang, respectively.