Better Spanish Emotion Recognition In-the-wild: Bringing Attention to Deep Spectrum Voice Analysis

TL;DR

The paper tackles Spanish emotion recognition in real-world settings for Socially Assistive Robots by evaluating DeepSpectrum variants on two Spanish datasets, ELRA-S0329 and EmoMatchSpanishDB. It introduces DS-AM, an attention-enhanced DeepSpectrum model built on a VGG-16 backbone, and compares DS-SVC, DS-FC, and DS-AM across datasets with 10-fold CV and cross-dataset testing to simulate in-the-wild conditions. DS-AM achieves state-of-the-art performance on both datasets, with cross-dataset experiments showing that models trained on EmoMatchSpanishDB generalize better to ELRA-S0329, underscoring dataset biases and the importance of speaker diversity. The work highlights the value of attention mechanisms in spectral-based emotion recognition for Spanish and points to future directions like expanding datasets and leveraging GAN-based augmentation to further improve robustness in-the-wild.

Abstract

Within the context of creating new Socially Assistive Robots, emotion recognition has become a key development factor, as it allows the robot to adapt to the user's emotional state in the wild. In this work, we focused on the analysis of two voice recording Spanish datasets: ELRA-S0329 and EmoMatchSpanishDB. Specifically, we centered our work in the paralanguage, e.~g. the vocal characteristics that go along with the message and clarifies the meaning. We proposed the use of the DeepSpectrum method, which consists of extracting a visual representation of the audio tracks and feeding them to a pretrained CNN model. For the classification task, DeepSpectrum is often paired with a Support Vector Classifier --DS-SVC--, or a Fully-Connected deep-learning classifier --DS-FC--. We compared the results of the DS-SVC and DS-FC architectures with the state-of-the-art (SOTA) for ELRA-S0329 and EmoMatchSpanishDB. Moreover, we proposed our own classifier based upon Attention Mechanisms, namely DS-AM. We trained all models against both datasets, and we found that our DS-AM model outperforms the SOTA models for the datasets and the SOTA DeepSpectrum architectures. Finally, we trained our DS-AM model in one dataset and tested it in the other, to simulate real-world conditions on how biased is the model to the dataset.

Figures (6)

• Figure 1: Schematic of a DeepSpectrum pipeline for audio classification. The audio signal is converted into a Mel spectrogram, which is then fed to a pretrained CNN backbone to extract features. The features are then classified using an available classifier.
• Figure 2: A typical DeepSpectrum-SVC (DS-SVC) pipeline for audio classification. The audio signal is converted into a Mel spectrogram, which is then fed to a pretrained CNN backbone to extract features. The features are then classified using a Support Vector Classifier.
• Figure 3: Another configuration of DeepSpectrum, DeepSpectrum-FC (DS-FN) where the features are extracted from the Mel spectrogram using a pretrained CNN backbone and then classified using a Fully-Connected deep-learning classifier.
• Figure 4: The DeepSpectrum-Attention Mechanisms (DS-AM) architecture. The figure shows: a) the general model which includes a modified DeepSpectrum-AM; and, b) the attention mechanism module.
• Figure 5: Confusion matrix for DeepSpectrum-AM over a) ELRA-S0329 dataset; and, b) EmoMatchSpanishDB dataset.