Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PHemoNet: A Multimodal Network for Physiological Signals

Eleonora Lopez, Aurelio Uncini, Danilo Comminiello

TL;DR

The proposed PHEmoNet, a fully hypercomplex network for multimodal emotion recognition from physiological signals, outperforms current state-of-the-art models on the MAHNOB-HCI dataset in classifying valence and arousal using electroencephalograms (EEGs) and peripheral physiological signals.

Abstract

Emotion recognition is essential across numerous fields, including medical applications and brain-computer interface (BCI). Emotional responses include behavioral reactions, such as tone of voice and body movement, and changes in physiological signals, such as the electroencephalogram (EEG). The latter are involuntary, thus they provide a reliable input for identifying emotions, in contrast to the former which individuals can consciously control. These signals reveal true emotional states without intentional alteration, thus increasing the accuracy of emotion recognition models. However, multimodal deep learning methods from physiological signals have not been significantly investigated. In this paper, we introduce PHemoNet, a fully hypercomplex network for multimodal emotion recognition from physiological signals. In detail, the architecture comprises modality-specific encoders and a fusion module. Both encoders and fusion modules are defined in the hypercomplex domain through parameterized hypercomplex multiplications (PHMs) that can capture latent relations between the different dimensions of each modality and between the modalities themselves. The proposed method outperforms current state-of-the-art models on the MAHNOB-HCI dataset in classifying valence and arousal using electroencephalograms (EEGs) and peripheral physiological signals. The code for this work is available at https://github.com/ispamm/MHyEEG.

PHemoNet: A Multimodal Network for Physiological Signals

TL;DR

The proposed PHEmoNet, a fully hypercomplex network for multimodal emotion recognition from physiological signals, outperforms current state-of-the-art models on the MAHNOB-HCI dataset in classifying valence and arousal using electroencephalograms (EEGs) and peripheral physiological signals.

Abstract

Emotion recognition is essential across numerous fields, including medical applications and brain-computer interface (BCI). Emotional responses include behavioral reactions, such as tone of voice and body movement, and changes in physiological signals, such as the electroencephalogram (EEG). The latter are involuntary, thus they provide a reliable input for identifying emotions, in contrast to the former which individuals can consciously control. These signals reveal true emotional states without intentional alteration, thus increasing the accuracy of emotion recognition models. However, multimodal deep learning methods from physiological signals have not been significantly investigated. In this paper, we introduce PHemoNet, a fully hypercomplex network for multimodal emotion recognition from physiological signals. In detail, the architecture comprises modality-specific encoders and a fusion module. Both encoders and fusion modules are defined in the hypercomplex domain through parameterized hypercomplex multiplications (PHMs) that can capture latent relations between the different dimensions of each modality and between the modalities themselves. The proposed method outperforms current state-of-the-art models on the MAHNOB-HCI dataset in classifying valence and arousal using electroencephalograms (EEGs) and peripheral physiological signals. The code for this work is available at https://github.com/ispamm/MHyEEG.
Paper Structure (11 sections, 2 equations, 1 figure, 1 table)

This paper contains 11 sections, 2 equations, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Related works
  3. Hypercomplex learning
  4. Emotion recognition
  5. Background
  6. Proposed Method
  7. Experimental results
  8. Dataset and preprocessing
  9. Experimental setup
  10. Results and discussion
  11. Conclusion

Figures (1)

  • Figure 1: PHemoNet. Fully hypercomplex architecture with encoders operating in different hypercomplex domains according to $n$ and a refined hypercomplex fusion module with $n_{\text{fusion}}=4$. Finally, a classification layer yileds the final output for valence/arousal prefiction.