Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Toward Robust Incomplete Multimodal Sentiment Analysis via Hierarchical Representation Learning

Mingcheng Li, Dingkang Yang, Yang Liu, Shunli Wang, Jiawei Chen, Shuaibing Wang, Jinjie Wei, Yue Jiang, Qingyao Xu, Xiaolu Hou, Mingyang Sun, Ziyun Qian, Dongliang Kou, Lihua Zhang

TL;DR

A fine-grained representation factorization module is proposed that sufficiently extracts valuable sentiment information by factorizing modality into sentiment-relevant and modality-specific representations through crossmodal translation and sentiment semantic reconstruction and a hierarchical mutual information maximization mechanism is introduced to incrementally maximize the mutual information between multi-scale representations to align and reconstruct the high-level semantics in the representations.

Abstract

Multimodal Sentiment Analysis (MSA) is an important research area that aims to understand and recognize human sentiment through multiple modalities. The complementary information provided by multimodal fusion promotes better sentiment analysis compared to utilizing only a single modality. Nevertheless, in real-world applications, many unavoidable factors may lead to situations of uncertain modality missing, thus hindering the effectiveness of multimodal modeling and degrading the model's performance. To this end, we propose a Hierarchical Representation Learning Framework (HRLF) for the MSA task under uncertain missing modalities. Specifically, we propose a fine-grained representation factorization module that sufficiently extracts valuable sentiment information by factorizing modality into sentiment-relevant and modality-specific representations through crossmodal translation and sentiment semantic reconstruction. Moreover, a hierarchical mutual information maximization mechanism is introduced to incrementally maximize the mutual information between multi-scale representations to align and reconstruct the high-level semantics in the representations. Ultimately, we propose a hierarchical adversarial learning mechanism that further aligns and adapts the latent distribution of sentiment-relevant representations to produce robust joint multimodal representations. Comprehensive experiments on three datasets demonstrate that HRLF significantly improves MSA performance under uncertain modality missing cases.

Toward Robust Incomplete Multimodal Sentiment Analysis via Hierarchical Representation Learning

TL;DR

A fine-grained representation factorization module is proposed that sufficiently extracts valuable sentiment information by factorizing modality into sentiment-relevant and modality-specific representations through crossmodal translation and sentiment semantic reconstruction and a hierarchical mutual information maximization mechanism is introduced to incrementally maximize the mutual information between multi-scale representations to align and reconstruct the high-level semantics in the representations.

Abstract

Multimodal Sentiment Analysis (MSA) is an important research area that aims to understand and recognize human sentiment through multiple modalities. The complementary information provided by multimodal fusion promotes better sentiment analysis compared to utilizing only a single modality. Nevertheless, in real-world applications, many unavoidable factors may lead to situations of uncertain modality missing, thus hindering the effectiveness of multimodal modeling and degrading the model's performance. To this end, we propose a Hierarchical Representation Learning Framework (HRLF) for the MSA task under uncertain missing modalities. Specifically, we propose a fine-grained representation factorization module that sufficiently extracts valuable sentiment information by factorizing modality into sentiment-relevant and modality-specific representations through crossmodal translation and sentiment semantic reconstruction. Moreover, a hierarchical mutual information maximization mechanism is introduced to incrementally maximize the mutual information between multi-scale representations to align and reconstruct the high-level semantics in the representations. Ultimately, we propose a hierarchical adversarial learning mechanism that further aligns and adapts the latent distribution of sentiment-relevant representations to produce robust joint multimodal representations. Comprehensive experiments on three datasets demonstrate that HRLF significantly improves MSA performance under uncertain modality missing cases.

Paper Structure

This paper contains 20 sections, 9 equations, 6 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multimodal Sentiment Analysis
  4. Factorized Representation Learning
  5. Knowledge Distillation
  6. Methodology
  7. Problem Formulation
  8. Overall Framework
  9. Fine-grained Representation Factorization
  10. Hierarchical Mutual Information Maximization
  11. Hierarchical Adversarial Learning
  12. Optimization Objectives
  13. Experiments
  14. Datasets and Evaluation Metrics
  15. Implementation Details
  16. ...and 5 more sections

Figures (6)

  • Figure 1: A case of incorrect prediction by the traditional model with missing modalities. The pink and yellow areas indicate intra- and inter-modality missingness, respectively.
  • Figure 2: The structure of our HRLF, which consists of three core components: Fine-grained Representation Factorization (FRF) module, Hierarchical Mutual Information (HMI) maximization mechanism, and Hierarchical Adversarial Learning (HAL) mechanism.
  • Figure 3: Comparison results of intra-modality missingness on IEMOCAP. We report on the F1 score metric for the happy, sad, angry, and neutral categories.
  • Figure 4: Comparison results of intra-modality missingness on (a) MOSI and (b) MOSEI.
  • Figure 5: Ablation results of intra-modality missingness case on the MOSI dataset.
  • ...and 1 more figures