Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MTPareto: A MultiModal Targeted Pareto Framework for Fake News Detection

Kaiying Yan, Moyang Liu, Yukun Liu, Ruibo Fu, Zhengqi Wen, Jianhua Tao, Xuefei Liu, Guanjun Li

TL;DR

The paper tackles multimodal fake news detection, where conflicting optimization signals across text, audio, and visuals hinder effective fusion. It introduces MTPareto, a hierarchical fusion framework that employs Targeted Pareto gradient integration to align objectives across three fusion levels, prioritizing all-modal information while controlling gradient conflicts. Key contributions include the three-level fusion architecture with attention-based modules, the TPareto training scheme with level-specific losses and gradient rules, and extensive experiments on FakeSV and FVC showing consistent accuracy gains and informative ablations. The approach demonstrates robust all-modal fusion and provides a transferable methodology for other multimodal tasks that face inter-modal optimization conflicts.

Abstract

Multimodal fake news detection is essential for maintaining the authenticity of Internet multimedia information. Significant differences in form and content of multimodal information lead to intensified optimization conflicts, hindering effective model training as well as reducing the effectiveness of existing fusion methods for bimodal. To address this problem, we propose the MTPareto framework to optimize multimodal fusion, using a Targeted Pareto(TPareto) optimization algorithm for fusion-level-specific objective learning with a certain focus. Based on the designed hierarchical fusion network, the algorithm defines three fusion levels with corresponding losses and implements all-modal-oriented Pareto gradient integration for each. This approach accomplishes superior multimodal fusion by utilizing the information obtained from intermediate fusion to provide positive effects to the entire process. Experiment results on FakeSV and FVC datasets show that the proposed framework outperforms baselines and the TPareto optimization algorithm achieves 2.40% and 1.89% accuracy improvement respectively.

MTPareto: A MultiModal Targeted Pareto Framework for Fake News Detection

TL;DR

The paper tackles multimodal fake news detection, where conflicting optimization signals across text, audio, and visuals hinder effective fusion. It introduces MTPareto, a hierarchical fusion framework that employs Targeted Pareto gradient integration to align objectives across three fusion levels, prioritizing all-modal information while controlling gradient conflicts. Key contributions include the three-level fusion architecture with attention-based modules, the TPareto training scheme with level-specific losses and gradient rules, and extensive experiments on FakeSV and FVC showing consistent accuracy gains and informative ablations. The approach demonstrates robust all-modal fusion and provides a transferable methodology for other multimodal tasks that face inter-modal optimization conflicts.

Abstract

Multimodal fake news detection is essential for maintaining the authenticity of Internet multimedia information. Significant differences in form and content of multimodal information lead to intensified optimization conflicts, hindering effective model training as well as reducing the effectiveness of existing fusion methods for bimodal. To address this problem, we propose the MTPareto framework to optimize multimodal fusion, using a Targeted Pareto(TPareto) optimization algorithm for fusion-level-specific objective learning with a certain focus. Based on the designed hierarchical fusion network, the algorithm defines three fusion levels with corresponding losses and implements all-modal-oriented Pareto gradient integration for each. This approach accomplishes superior multimodal fusion by utilizing the information obtained from intermediate fusion to provide positive effects to the entire process. Experiment results on FakeSV and FVC datasets show that the proposed framework outperforms baselines and the TPareto optimization algorithm achieves 2.40% and 1.89% accuracy improvement respectively.
Paper Structure (18 sections, 4 equations, 1 figure, 4 tables)

This paper contains 18 sections, 4 equations, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Multimodal Hierarchical Fusion Architecture
  3. Overview of Model Architecture
  4. Hierarchical Fusion Modules
  5. Targeted Pareto Optimization Algorithm
  6. Pareto Integration Preliminaries
  7. Targeted Pareto Integration in Hierarchical-Fusion-based Multimodal Learning
  8. Experiment
  9. Datasets and Evaluation
  10. Datasets.
  11. Experimental Setup.
  12. Results
  13. Comparison with Baselines
  14. Ablation Study
  15. Exploration Study
  16. ...and 3 more sections

Figures (1)

  • Figure 1: Illustration of the MTPareto framework.