Unifying EEG and Speech for Emotion Recognition: A Two-Step Joint Learning Framework for Handling Missing EEG Data During Inference

TL;DR

This work tackles reliable automatic emotion recognition by leveraging EEG during training to bolster speech-based AER, addressing the practical constraint of EEG unavailability at inference. It introduces a two-step JMML framework: first, intra-modal learning via JEC-SSL to capture both class-specific and shared emotion characteristics within each modality, and second, inter-modal learning with an extended DCC-CAE (E-DCC-CAE) to align EEG and speech in a correlated joint space. The approach handles non-parallel bimodal data and missing modality at inference by enabling cross reconstruction, and experiments show meaningful gains, especially in speech recognition, with EEG contributing during training but not required for testing. Overall, JMML demonstrates that speech representations can be enriched through EEG priors to achieve robust AER in realistic scenarios where EEG data are hard to obtain.

Abstract

Computer interfaces are advancing towards using multi-modalities to enable better human-computer interactions. The use of automatic emotion recognition (AER) can make the interactions natural and meaningful thereby enhancing the user experience. Though speech is the most direct and intuitive modality for AER, it is not reliable because it can be intentionally faked by humans. On the other hand, physiological modalities like EEG, are more reliable and impossible to fake. However, use of EEG is infeasible for realistic scenarios usage because of the need for specialized recording setup. In this paper, one of our primary aims is to ride on the reliability of the EEG modality to facilitate robust AER on the speech modality. Our approach uses both the modalities during training to reliably identify emotion at the time of inference, even in the absence of the more reliable EEG modality. We propose, a two-step joint multi-modal learning approach (JMML) that exploits both the intra- and inter- modal characteristics to construct emotion embeddings that enrich the performance of AER. In the first step, using JEC-SSL, intra-modal learning is done independently on the individual modalities. This is followed by an inter-modal learning using the proposed extended variant of deep canonically correlated cross-modal autoencoder (E-DCC-CAE). The approach learns the joint properties of both the modalities by mapping them into a common representation space, such that the modalities are maximally correlated. These emotion embeddings, hold properties of both the modalities there by enhancing the performance of ML classifier used for AER. Experimental results show the efficacy of the proposed approach. To best of our knowledge, this is the first attempt to combine speech and EEG with joint multi-modal learning approach for reliable AER.

Figures (5)

• Figure 1: Two class emotion representation ($E_1$ and $E_2$) with shared characteristics ($E_1 \cap E_2$).
• Figure 2: Sample representation of the shared characteristics ($H_y$, $A_y$) and class specific characteristics ($H_x$, $A_x$) for two emotion classes, Anger ($A$) and Happy ($H$) in the valence-arousal emotion space.
• Figure 3: System Overview.
• Figure 4: Joint Emotion Class Learning ( jecl) for $C$ classes, followed by mbpls based ssl ( jec- ssl). The $j^{th}$ class learns from all other classes through its similarity latent space as well as its independent latent space.
• Figure 5: Joint Multi-modal Learning ( jmml) for bimodal data; s-rec and x-rec represents the reconstructed version of self and other respective modality