Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hierarchical Hypercomplex Network for Multimodal Emotion Recognition

Eleonora Lopez, Aurelio Uncini, Danilo Comminiello

TL;DR

A fully hypercomplex network with a hierarchical learning structure to fully capture correlations and surpasses state-of-the-art models on the MAHNOB-HCI dataset for emotion recognition, specifically in classifying valence and arousal from electroencephalograms (EEGs) and peripheral physiological signals.

Abstract

Emotion recognition is relevant in various domains, ranging from healthcare to human-computer interaction. Physiological signals, being beyond voluntary control, offer reliable information for this purpose, unlike speech and facial expressions which can be controlled at will. They reflect genuine emotional responses, devoid of conscious manipulation, thereby enhancing the credibility of emotion recognition systems. Nonetheless, multimodal emotion recognition with deep learning models remains a relatively unexplored field. In this paper, we introduce a fully hypercomplex network with a hierarchical learning structure to fully capture correlations. Specifically, at the encoder level, the model learns intra-modal relations among the different channels of each input signal. Then, a hypercomplex fusion module learns inter-modal relations among the embeddings of the different modalities. The main novelty is in exploiting intra-modal relations by endowing the encoders with parameterized hypercomplex convolutions (PHCs) that thanks to hypercomplex algebra can capture inter-channel interactions within single modalities. Instead, the fusion module comprises parameterized hypercomplex multiplications (PHMs) that can model inter-modal correlations. The proposed architecture surpasses state-of-the-art models on the MAHNOB-HCI dataset for emotion recognition, specifically in classifying valence and arousal from electroencephalograms (EEGs) and peripheral physiological signals. The code of this study is available at https://github.com/ispamm/MHyEEG.

Hierarchical Hypercomplex Network for Multimodal Emotion Recognition

TL;DR

A fully hypercomplex network with a hierarchical learning structure to fully capture correlations and surpasses state-of-the-art models on the MAHNOB-HCI dataset for emotion recognition, specifically in classifying valence and arousal from electroencephalograms (EEGs) and peripheral physiological signals.

Abstract

Emotion recognition is relevant in various domains, ranging from healthcare to human-computer interaction. Physiological signals, being beyond voluntary control, offer reliable information for this purpose, unlike speech and facial expressions which can be controlled at will. They reflect genuine emotional responses, devoid of conscious manipulation, thereby enhancing the credibility of emotion recognition systems. Nonetheless, multimodal emotion recognition with deep learning models remains a relatively unexplored field. In this paper, we introduce a fully hypercomplex network with a hierarchical learning structure to fully capture correlations. Specifically, at the encoder level, the model learns intra-modal relations among the different channels of each input signal. Then, a hypercomplex fusion module learns inter-modal relations among the embeddings of the different modalities. The main novelty is in exploiting intra-modal relations by endowing the encoders with parameterized hypercomplex convolutions (PHCs) that thanks to hypercomplex algebra can capture inter-channel interactions within single modalities. Instead, the fusion module comprises parameterized hypercomplex multiplications (PHMs) that can model inter-modal correlations. The proposed architecture surpasses state-of-the-art models on the MAHNOB-HCI dataset for emotion recognition, specifically in classifying valence and arousal from electroencephalograms (EEGs) and peripheral physiological signals. The code of this study is available at https://github.com/ispamm/MHyEEG.
Paper Structure (11 sections, 2 equations, 2 figures, 3 tables)

This paper contains 11 sections, 2 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related works
  3. Method
  4. Parameterized Hypercomplex Networks
  5. Hierarchical Hypercomplex Model
  6. Experimental results
  7. Dataset and preprocessing
  8. Experimental setup
  9. Results and discussion
  10. Ablation studies
  11. Conclusion

Figures (2)

  • Figure 1: H2 model. The encoder learns enriched modality-specific embeddings by exploiting inter-channel relations within single modalities. The hypercomplex fusion module learns inter-modal correlations. Finally, a fully-connected layer produces the prediction for arousal/valence. In PHC and PHM layers, the smaller blocks indicate the number of submatrices of the weight matrix ${\mathbf W}$ following eq.\ref{['eq:matirx']}, which are shared among input dimensions and allow learning relations among them.
  • Figure 2: t-SNE feature visualization.