Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bridging Cognition and Emotion: Empathy-Driven Multimodal Misinformation Detection

Zihan Wang, Lu Yuan, Zhengxuan Zhang, Qing Zhao

TL;DR

This work tackles misinformation detection by integrating cognitive and emotional empathy through a Dual-Aspect Empathy Framework (DAE) that analyzes creator strategies and simulates reader responses using Large Language Models. The method introduces an empathy-aware comment generation and filtering pipeline, multi-level feature extraction with cross-modal fusion, and a dual-empathy representation that encodes both cognitive alignment and emotional gaps. Empirical results on publicly available multimodal datasets show that DAE consistently outperforms strong baselines, with ablations confirming the importance of text, reader comments, and both empathy dimensions. The proposed human-centric approach offers a new paradigm for interpretable and robust multimodal misinformation detection, with potential applicability to real-time and cross-cultural settings.

Abstract

In the digital era, social media has become a major conduit for information dissemination, yet it also facilitates the rapid spread of misinformation. Traditional misinformation detection methods primarily focus on surface-level features, overlooking the crucial roles of human empathy in the propagation process. To address this gap, we propose the Dual-Aspect Empathy Framework (DAE), which integrates cognitive and emotional empathy to analyze misinformation from both the creator and reader perspectives. By examining creators' cognitive strategies and emotional appeals, as well as simulating readers' cognitive judgments and emotional responses using Large Language Models (LLMs), DAE offers a more comprehensive and human-centric approach to misinformation detection. Moreover, we further introduce an empathy-aware filtering mechanism to enhance response authenticity and diversity. Experimental results on benchmark datasets demonstrate that DAE outperforms existing methods, providing a novel paradigm for multimodal misinformation detection.

Bridging Cognition and Emotion: Empathy-Driven Multimodal Misinformation Detection

TL;DR

This work tackles misinformation detection by integrating cognitive and emotional empathy through a Dual-Aspect Empathy Framework (DAE) that analyzes creator strategies and simulates reader responses using Large Language Models. The method introduces an empathy-aware comment generation and filtering pipeline, multi-level feature extraction with cross-modal fusion, and a dual-empathy representation that encodes both cognitive alignment and emotional gaps. Empirical results on publicly available multimodal datasets show that DAE consistently outperforms strong baselines, with ablations confirming the importance of text, reader comments, and both empathy dimensions. The proposed human-centric approach offers a new paradigm for interpretable and robust multimodal misinformation detection, with potential applicability to real-time and cross-cultural settings.

Abstract

In the digital era, social media has become a major conduit for information dissemination, yet it also facilitates the rapid spread of misinformation. Traditional misinformation detection methods primarily focus on surface-level features, overlooking the crucial roles of human empathy in the propagation process. To address this gap, we propose the Dual-Aspect Empathy Framework (DAE), which integrates cognitive and emotional empathy to analyze misinformation from both the creator and reader perspectives. By examining creators' cognitive strategies and emotional appeals, as well as simulating readers' cognitive judgments and emotional responses using Large Language Models (LLMs), DAE offers a more comprehensive and human-centric approach to misinformation detection. Moreover, we further introduce an empathy-aware filtering mechanism to enhance response authenticity and diversity. Experimental results on benchmark datasets demonstrate that DAE outperforms existing methods, providing a novel paradigm for multimodal misinformation detection.

Paper Structure

This paper contains 24 sections, 20 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Misinformation Detection
  4. Empathy in Social Computing
  5. Method
  6. Task Definition
  7. Comment Generation and Filtering
  8. Demographic-Based Comment Generation
  9. Empathy-Driven Comment Filtering
  10. Feature Extraction and Encoding
  11. Feature Enhancement and Interaction
  12. Dual-Aspect Empathy Feature Construction
  13. Basic Feature Extraction
  14. Empathy-aware Fusion
  15. Emotion Gap Computing
  16. ...and 9 more sections

Figures (5)

  • Figure 1: An Example of Misinformation.
  • Figure 2: Overview of Our Proposed Dual-Aspect Empathy Framework (DAE)
  • Figure 3: Results of Ablation Study
  • Figure 4: Several Cases in the Test Set of Datasets
  • Figure 5: Results of Parameter Analysis