EMOVOME: A Dataset for Emotion Recognition in Spontaneous Real-Life Speech

TL;DR

This work tackles the lack of spontaneous, real-life SER data by introducing EMOVOME, the first public dataset of spontaneous emotions from real WhatsApp conversations in Spanish. It evaluates speaker-independent SER using both acoustic baselines (eGeMAPS) and pre-trained model embeddings (e.g., UniSpeech-SAT-Large, HuBERT, Wav2Vec2 variants), including a fusion with eGeMAPS, across valence, arousal, and emotion categories. EMOVOME shows lower performance than acted datasets like RAVDESS and elicited IEMOCAP, but pre-trained embeddings substantially improve results, with UniSpeech-SAT-Large achieving the best valence (61.64% UA) and arousal (55.57% UA) on EMOVOME, and 42.58% UA for emotion categories. The study also reveals annotator-driven effects and gender fairness patterns, highlighting that combining expert and non-expert labels can enhance fairness while acknowledging subjective labeling in naturalistic data, underscoring the need for real-world evaluation in SER advances.

Abstract

Spontaneous datasets for Speech Emotion Recognition (SER) are scarce and frequently derived from laboratory environments or staged scenarios, such as TV shows, limiting their application in real-world contexts. We developed and publicly released the Emotional Voice Messages (EMOVOME) dataset, including 999 voice messages from real conversations of 100 Spanish speakers on a messaging app, labeled in continuous and discrete emotions by expert and non-expert annotators. We evaluated speaker-independent SER models using acoustic features as baseline and transformer-based models. We compared the results with reference datasets including acted and elicited speech, and analyzed the influence of annotators and gender fairness. The pre-trained UniSpeech-SAT-Large model achieved the highest results, 61.64% and 55.57% Unweighted Accuracy (UA) for 3-class valence and arousal prediction respectively on EMOVOME, a 10% improvement over baseline models. For the emotion categories, 42.58% UA was obtained. EMOVOME performed lower than the acted RAVDESS dataset. The elicited IEMOCAP dataset also outperformed EMOVOME in predicting emotion categories, while similar results were obtained in valence and arousal. EMOVOME outcomes varied with annotator labels, showing better results and fairness when combining expert and non-expert annotations. This study highlights the gap between controlled and real-life scenarios, supporting further advancements in recognizing genuine emotions.

Figures (7)

• Figure 1: Overview of the methodology.
• Figure 2: Distribution of audio samples based on their arousal and valence labels. The circle area is proportional to the number of samples in each group.
• Figure 3: Valence and arousal ratings for each emotion. The diameter represents the number of samples using a logarithmic scale
• Figure 4: Cross-validation results for the three methods implemented (eGeMAPS, Embeddings and Emb+eGeMAPS) across the different datasets (EMOVOME, IEMOCAP and RAVDESS) and emotion labels (valence, arousal and categories of emotions).
• Figure 5: Confusion matrix for the test samples for valence and arousal prediction on EMOVOME, for the different raters.