Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning Noise-Robust Joint Representation for Multimodal Emotion Recognition under Incomplete Data Scenarios

Qi Fan, Haolin Zuo, Rui Liu, Zheng Lian, Guanglai Gao

TL;DR

A novel noise-robust MER model is introduced that effectively learns robust multimodal joint representations from noisy data and enables the exploration of an entirely new type of incomplete data condition, which is impossible with existing methods.

Abstract

Multimodal emotion recognition (MER) in practical scenarios is significantly challenged by the presence of missing or incomplete data across different modalities. To overcome these challenges, researchers have aimed to simulate incomplete conditions during the training phase to enhance the system's overall robustness. Traditional methods have often involved discarding data or substituting data segments with zero vectors to approximate these incompletenesses. However, such approaches neither accurately represent real-world conditions nor adequately address the issue of noisy data availability. For instance, a blurry image cannot be simply replaced with zero vectors, while still retaining information. To tackle this issue and develop a more precise MER system, we introduce a novel noise-robust MER model that effectively learns robust multimodal joint representations from noisy data. This approach includes two pivotal components: firstly, a noise scheduler that adjusts the type and level of noise in the data to emulate various realistic incomplete situations. Secondly, a Variational AutoEncoder (VAE)-based module is employed to reconstruct these robust multimodal joint representations from the noisy inputs. Notably, the introduction of the noise scheduler enables the exploration of an entirely new type of incomplete data condition, which is impossible with existing methods. Extensive experimental evaluations on the benchmark datasets IEMOCAP and CMU-MOSEI demonstrate the effectiveness of the noise scheduler and the excellent performance of our proposed model. Our project is publicly available on https://github.com/WooyoohL/Noise-robust_MER.

Learning Noise-Robust Joint Representation for Multimodal Emotion Recognition under Incomplete Data Scenarios

TL;DR

A novel noise-robust MER model is introduced that effectively learns robust multimodal joint representations from noisy data and enables the exploration of an entirely new type of incomplete data condition, which is impossible with existing methods.

Abstract

Multimodal emotion recognition (MER) in practical scenarios is significantly challenged by the presence of missing or incomplete data across different modalities. To overcome these challenges, researchers have aimed to simulate incomplete conditions during the training phase to enhance the system's overall robustness. Traditional methods have often involved discarding data or substituting data segments with zero vectors to approximate these incompletenesses. However, such approaches neither accurately represent real-world conditions nor adequately address the issue of noisy data availability. For instance, a blurry image cannot be simply replaced with zero vectors, while still retaining information. To tackle this issue and develop a more precise MER system, we introduce a novel noise-robust MER model that effectively learns robust multimodal joint representations from noisy data. This approach includes two pivotal components: firstly, a noise scheduler that adjusts the type and level of noise in the data to emulate various realistic incomplete situations. Secondly, a Variational AutoEncoder (VAE)-based module is employed to reconstruct these robust multimodal joint representations from the noisy inputs. Notably, the introduction of the noise scheduler enables the exploration of an entirely new type of incomplete data condition, which is impossible with existing methods. Extensive experimental evaluations on the benchmark datasets IEMOCAP and CMU-MOSEI demonstrate the effectiveness of the noise scheduler and the excellent performance of our proposed model. Our project is publicly available on https://github.com/WooyoohL/Noise-robust_MER.
Paper Structure (21 sections, 7 equations, 2 figures, 5 tables, 1 algorithm)

This paper contains 21 sections, 7 equations, 2 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. NMER: Methodology
  3. Overall Architecture
  4. Noise Scheduler
  5. Rationale and Noise Selection
  6. Noisy Embedding Construction
  7. Noise Intensity Calculation
  8. VAE-based Network
  9. Loss Functions
  10. Experiments and Results
  11. Data Setup
  12. New Incomplete Condition
  13. Experimental Setup
  14. Comparison and Ablation Study
  15. Results
  16. ...and 6 more sections

Figures (2)

  • Figure 1: The incompleteness of multimodal data in realistic scenarios poses a significant challenge to multimodal emotion recognition.
  • Figure 2: The structure of our NMER model, including noise scheduler, VAE-based network, and the classifier. $\mathcal{L}_{inv}$ stands the invariant loss, $\mathcal{L}_{gen}$ is the generation loss, $\mathcal{L}_{cls}$ refers to the classification loss.