Interpretable Multimodal Misinformation Detection with Logic Reasoning

TL;DR

A novel logic-based neural model for multimodal misinformation detection which integrates interpretable logic clauses to express the reasoning process of the target task and is generalizable across diverse misinformation sources.

Abstract

Multimodal misinformation on online social platforms is becoming a critical concern due to increasing credibility and easier dissemination brought by multimedia content, compared to traditional text-only information. While existing multimodal detection approaches have achieved high performance, the lack of interpretability hinders these systems' reliability and practical deployment. Inspired by NeuralSymbolic AI which combines the learning ability of neural networks with the explainability of symbolic learning, we propose a novel logic-based neural model for multimodal misinformation detection which integrates interpretable logic clauses to express the reasoning process of the target task. To make learning effective, we parameterize symbolic logical elements using neural representations, which facilitate the automatic generation and evaluation of meaningful logic clauses. Additionally, to make our framework generalizable across diverse misinformation sources, we introduce five meta-predicates that can be instantiated with different correlations. Results on three public datasets (Twitter, Weibo, and Sarcasm) demonstrate the feasibility and versatility of our model.

Figures (6)

• Figure 1: Examples of explanations generated by attention map, multi-view, and our proposed Neural-Symbolic-based method for a rumor sample in Twitter dataset. For (c) and (d), a higher value indicates a higher probability of being detected as a rumor.
• Figure 2: The core architecture of the proposed interpretable multimodal misinformation detection framework based on logic reasoning (LogicDM). Textual nodes are fully connected to visual nodes but we only visualize edges between one textual node and visual nodes for ease of illustration.
• Figure 3: Examples of derived clauses and related constants. For (c) and (d), we translate the text from Chinese to English.
• Figure 4: The influence of the number of correlations $g$ for dynamic predicate representation.
• Figure 5: The influence of rate $\beta$ for logic clause generation.