Switchboard-Affect: Emotion Perception Labels from Conversational Speech

TL;DR

This work addresses the gap between speech emotion recognition (SER) models and real-world spontaneous speech by creating Switchboard-Affect (SWB-Affect), a crowd-annotated set of naturalistic emotion perception labels for the Switchboard corpus, including 10 categorical emotions and 3 dimensional attributes (activation, valence, dominance). It details rigorous annotator training, a gold-set-based certification, and quality controls, releasing 10,000 segments (~25 hours) with both consensus and per-annotator data. Analyses reveal that fear is often detectable lexically while happiness and related states rely more on paralinguistic cues, and demonstrate that model performance on naturalistic data is domain-sensitive, with anger being particularly challenging. The dataset enables richer evaluation of SER in natural conversational contexts and points toward more robust, context-aware emotion understanding in real-world applications, while also highlighting ethical considerations and the need for diverse annotators and contexts.

Abstract

Understanding the nuances of speech emotion dataset curation and labeling is essential for assessing speech emotion recognition (SER) model potential in real-world applications. Most training and evaluation datasets contain acted or pseudo-acted speech (e.g., podcast speech) in which emotion expressions may be exaggerated or otherwise intentionally modified. Furthermore, datasets labeled based on crowd perception often lack transparency regarding the guidelines given to annotators. These factors make it difficult to understand model performance and pinpoint necessary areas for improvement. To address this gap, we identified the Switchboard corpus as a promising source of naturalistic conversational speech, and we trained a crowd to label the dataset for categorical emotions (anger, contempt, disgust, fear, sadness, surprise, happiness, tenderness, calmness, and neutral) and dimensional attributes (activation, valence, and dominance). We refer to this label set as Switchboard-Affect (SWB-Affect). In this work, we present our approach in detail, including the definitions provided to annotators and an analysis of the lexical and paralinguistic cues that may have played a role in their perception. In addition, we evaluate state-of-the-art SER models, and we find variable performance across the emotion categories with especially poor generalization for anger. These findings underscore the importance of evaluation with datasets that capture natural affective variations in speech. We release the labels for SWB-Affect to enable further analysis in this domain.

Figures (3)

• Figure 1: Distribution of affect labels
• Figure 2: Co-occurence of affect labels
• Figure 3: Relationship between dimensional and categorical labels