EMO-SUPERB: An In-depth Look at Speech Emotion Recognition

TL;DR

This work identifies reproducibility and data-leakage issues in Speech Emotion Recognition and presents EMO-SUPERB, a community-driven benchmark that evaluates 15 SSLMs across six SER datasets with a public leaderboard.It introduces an end-to-end pipeline combining a GPT-based relabeling workflow, a comprehensive SSLM-based evaluation codebase, unified dataset partitions, and a live leaderboard to accelerate SER research and collaboration.Typed descriptions, though rare at about $2.58\%$, are relabeled by ChatGPT to produce distribution-like targets, yielding an average relative improvement of $3.08\%$ across 16 models, with CPC reaching up to $9.45\%$ gains on MSP-PODCAST.Analyses show SSLMs outperform traditional features, with XLS-R-1B often achieving the best average performance while emphasizing shallow layers for emotional encoding; ChatGPT relabeling emerges as a promising avenue to unlock natural-language annotations.The open-source ecosystem, including data partitions, relabeled datasets, code, and artifacts, aims to improve reproducibility and fairness in SER, with planned future work on broader languages, calibration, and bias assessment.

Abstract

Speech emotion recognition (SER) is a pivotal technology for human-computer interaction systems. However, 80.77% of SER papers yield results that cannot be reproduced. We develop EMO-SUPERB, short for EMOtion Speech Universal PERformance Benchmark, which aims to enhance open-source initiatives for SER. EMO-SUPERB includes a user-friendly codebase to leverage 15 state-of-the-art speech self-supervised learning models (SSLMs) for exhaustive evaluation across six open-source SER datasets. EMO-SUPERB streamlines result sharing via an online leaderboard, fostering collaboration within a community-driven benchmark and thereby enhancing the development of SER. On average, 2.58% of annotations are annotated using natural language. SER relies on classification models and is unable to process natural languages, leading to the discarding of these valuable annotations. We prompt ChatGPT to mimic annotators, comprehend natural language annotations, and subsequently re-label the data. By utilizing labels generated by ChatGPT, we consistently achieve an average relative gain of 3.08% across all settings.

Figures (11)

• Figure 1: Demonstration for the EMO-SUPERB platform: Developers design and evaluate SER models using our standardized dataset partition files and evaluation criteria. Developers then contribute these prediction results to the online leaderboard, enriching the benchmark database and enabling comparative analyses with other SER models. Finally, developers harness the visualization and statistical tools on the website to compare performance, gathering invaluable insights for future works. From the user's standpoint, they can upload datasets and select appropriate models tailored to their individual applications.
• Figure 2: Labeling process using ChatGPT. Three inputs are Typed description, Reference distribution, and Prompt. Two outputs are Reason and Adjusted distribution. Notice that the reference distribution is calculated by the number of votes for emotion classes. In the raw annotations of an example, there are instances of disgust, contempt, fear, neutrality, and happiness (*6), resulting in values of 0.6 for happiness and 0.1 for each of the remaining appearing emotions.
• Figure 3: Illustration of SSLM-based SER
• Figure 4: The layerwise weights analysis.
• Figure 5: Original and adjusted distributions. The original distribution, determined by tallying the votes for each emotion class, is compared with the adjusted distribution resulting from ChatGPT's re-labeling. In the raw annotations of the example, there are instances of disgust, contempt, fear, neutrality, and happiness (*6), resulting in values of 0.6 for happiness and 0.1 for each of the remaining emotions.