MATER: Multi-level Acoustic and Textual Emotion Representation for Interpretable Speech Emotion Recognition

TL;DR

The paper tackles speech emotion recognition in naturalistic conditions by introducing MATER, a hierarchical framework that fuses word-, utterance-, and embedding-level acoustic-textual features. An uncertainty-aware ensemble mitigates annotator disagreement and improves robustness, achieving a final Macro-F1 of $41.01\%$ and an average CCC of $0.5928$, with valence prediction reaching $0.6941$. Key findings show word-level syntactic-prosodic cues are highly informative for categorization, while embedding-level, domain-adapted representations aid generalization; acoustic cues predominantly drive arousal and dominance. Overall, MATER demonstrates strong performance and interpretability on the SERNC MSP-Podcast dataset, offering a practical, robust blueprint for real-world SER under variability and annotation uncertainty.

Abstract

This paper presents our contributions to the Speech Emotion Recognition in Naturalistic Conditions (SERNC) Challenge, where we address categorical emotion recognition and emotional attribute prediction. To handle the complexities of natural speech, including intra- and inter-subject variability, we propose Multi-level Acoustic-Textual Emotion Representation (MATER), a novel hierarchical framework that integrates acoustic and textual features at the word, utterance, and embedding levels. By fusing low-level lexical and acoustic cues with high-level contextualized representations, MATER effectively captures both fine-grained prosodic variations and semantic nuances. Additionally, we introduce an uncertainty-aware ensemble strategy to mitigate annotator inconsistencies, improving robustness in ambiguous emotional expressions. MATER ranks fourth in both tasks with a Macro-F1 of 41.01% and an average CCC of 0.5928, securing second place in valence prediction with an impressive CCC of 0.6941.

Figures (4)

• Figure 1: The MATER framework that integrates word-, utterance-, and embedding-level features to construct robust and context-aware emotion representations.
• Figure 2: Emotional category distribution and annotator agreement in the train and development sets for Task 1. Bars represent sample proportions, highlighting class imbalance, while lines indicate average agreement ratios. Shaded areas denote standard deviation, illustrating consensus inconsistencies.
• Figure 3: Feature aggregation of MATER. Embedding-level features are aggregated through the Perceiver, utterance-level features use PLE with a linear layer, and word-level features use a two-layer LSTM. The concatenated features are passed to a linear classifier for Task 1 and a linear regressor for Task 2.
• Figure 4: Performance comparison across feature levels for Task 1 (Macro-F1) and Task 2 (average CCC for valence, arousal, and dominance) on the development set.