MSF-SER: Enriching Acoustic Modeling with Multi-Granularity Semantics for Speech Emotion Recognition

TL;DR

MSF-SER tackles the limitations of relying solely on global transcripts for dimensional speech emotion recognition by introducing three semantic granularities: Local Emphasized Semantics ($LES$), Global Semantics ($GS$), and Extended Semantics ($ES$). It combines these with a WavLM-Large acoustic backbone via an intra-modal gated fusion and a cross-modal FiLM-modulated Mixture-of-Experts (FM-MOE), enabling dimension-specific cross-modal routing for $V$, $A$, and $D$. The approach leverages LEMF for emphasis detection, Whisper-ASR + RoBERTa-Large for global semantics, and Kimi-Audio + RoBERTa-Large for extended semantics, all fused through FiLM and MoE to guide acoustic learning. Evaluations on MSP-Podcast and IEMOCAP show consistent improvements in CCC scores, with LES/GS boosting valence and arousal while ES enhances dominance, demonstrating the effectiveness and generalizability of multi-granularity semantic fusion for SER. The work suggests significant practical impact for robust real-world affective systems and opens avenues for cross-lingual and multimodal extensions.

Abstract

Continuous dimensional speech emotion recognition captures affective variation along valence, arousal, and dominance, providing finer-grained representations than categorical approaches. Yet most multimodal methods rely solely on global transcripts, leading to two limitations: (1) all words are treated equally, overlooking that emphasis on different parts of a sentence can shift emotional meaning; (2) only surface lexical content is represented, lacking higher-level interpretive cues. To overcome these issues, we propose MSF-SER (Multi-granularity Semantic Fusion for Speech Emotion Recognition), which augments acoustic features with three complementary levels of textual semantics--Local Emphasized Semantics (LES), Global Semantics (GS), and Extended Semantics (ES). These are integrated via an intra-modal gated fusion and a cross-modal FiLM-modulated lightweight Mixture-of-Experts (FM-MOE). Experiments on MSP-Podcast and IEMOCAP show that MSF-SER consistently improves dimensional prediction, demonstrating the effectiveness of enriched semantic fusion for SER.

Figures (2)

• Figure 1: Our proposed framework adopts acoustic features as the backbone, while three levels of semantic features are incrementally incorporated as auxiliary signals. An intra-modal gating mechanism fuses global and local semantics, and cross-modal integration between acoustic and semantic representations is achieved through FM-MOE.
• Figure 2: Illustration of emphasis detection by LEMF. The top figure shows the log-F0 curve over time with the top-3 emphasized words highlighted, while the bottom figure shows the corresponding energy-time curve.