Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multimodal Interaction Modeling via Self-Supervised Multi-Task Learning for Review Helpfulness Prediction

HongLin Gong, Mengzhao Jia, Liqiang Jing

TL;DR

This work proposes an auto-generated scheme based on multi-task learning to generate pseudo labels that surpasses previous textual and multimodal baseline models on two widely accessible benchmark datasets, providing a solution to the MRHP problem.

Abstract

In line with the latest research, the task of identifying helpful reviews from a vast pool of user-generated textual and visual data has become a prominent area of study. Effective modal representations are expected to possess two key attributes: consistency and differentiation. Current methods designed for Multimodal Review Helpfulness Prediction (MRHP) face limitations in capturing distinctive information due to their reliance on uniform multimodal annotation. The process of adding varied multimodal annotations is not only time-consuming but also labor-intensive. To tackle these challenges, we propose an auto-generated scheme based on multi-task learning to generate pseudo labels. This approach allows us to simultaneously train for the global multimodal interaction task and the separate cross-modal interaction subtasks, enabling us to learn and leverage both consistency and differentiation effectively. Subsequently, experimental results validate the effectiveness of pseudo labels, and our approach surpasses previous textual and multimodal baseline models on two widely accessible benchmark datasets, providing a solution to the MRHP problem.

Multimodal Interaction Modeling via Self-Supervised Multi-Task Learning for Review Helpfulness Prediction

TL;DR

This work proposes an auto-generated scheme based on multi-task learning to generate pseudo labels that surpasses previous textual and multimodal baseline models on two widely accessible benchmark datasets, providing a solution to the MRHP problem.

Abstract

In line with the latest research, the task of identifying helpful reviews from a vast pool of user-generated textual and visual data has become a prominent area of study. Effective modal representations are expected to possess two key attributes: consistency and differentiation. Current methods designed for Multimodal Review Helpfulness Prediction (MRHP) face limitations in capturing distinctive information due to their reliance on uniform multimodal annotation. The process of adding varied multimodal annotations is not only time-consuming but also labor-intensive. To tackle these challenges, we propose an auto-generated scheme based on multi-task learning to generate pseudo labels. This approach allows us to simultaneously train for the global multimodal interaction task and the separate cross-modal interaction subtasks, enabling us to learn and leverage both consistency and differentiation effectively. Subsequently, experimental results validate the effectiveness of pseudo labels, and our approach surpasses previous textual and multimodal baseline models on two widely accessible benchmark datasets, providing a solution to the MRHP problem.
Paper Structure (18 sections, 21 equations, 4 figures, 4 tables)

This paper contains 18 sections, 21 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Formulation
  4. METHODOLOGY
  5. Modality-specific Feature Extraction
  6. Interaction-aware Consistency Modeling
  7. Global Interaction-aware Helpfulness Scoring
  8. Fine-grained Contribution-guided Helpfulness Measurement
  9. EXPERIMENT
  10. Experimental Settings
  11. On Model Comparison
  12. On Ablation Study
  13. On Case Study
  14. Conclusion
  15. Conclusion
  16. ...and 3 more sections

Figures (4)

  • Figure 1: Illustration of the consistent and different correlations between the textual and visual modalities.
  • Figure 2: Illustration of the proposed MM-SS scheme. It consists of four modules: (a) Modality-specific Feature Extration, (b) Interaction-aware Consistency Modeling, (c) Global Interaction-aware Helpfulness Scoring, and (d) Fine-grained Contribution-guided Helpfulness Measurement.
  • Figure 3: Pseudo label generation example, we can conclude the relationship: $y_{s}=y_{g}+\delta_{gs}$, where $\delta_{gs}$ is offset, and $s\in\{p_{t}r_{t},p_{v}r_{v},p_{t}r_{v},p_{v}r_{t}, r_{t}r_{v}\}$.
  • Figure 4: Case study for the MM-SS on Amazon-MRHP. The "H" is human-annotated, "M" is our model prediction, "O" is our model prediction without SSP-Labels, and others are auto-generated SSP-Labels.