Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MSAC: Multiple Speech Attribute Control Method for Reliable Speech Emotion Recognition

Yu Pan, Yuguang Yang, Yuheng Huang, Jixun Yao, Jingjing Yin, Yanni Hu, Heng Lu, Lei Ma, Jianjun Zhao

TL;DR

This work tackles the reliability of speech emotion recognition under semantic data shifts by introducing MSAC-SERNet, a unified framework that jointly handles single-corpus and cross-corpus SER. It introduces Multiple Speech Attribute Control (MSAC), a paradigm that models correlations among speech attributes and mitigates emotion-agnostic representations, aided by FBanks inputs and an AM-Softmax-based loss. The approach achieves state-of-the-art results on six public corpora, demonstrating improved recognition, generalization, and reliability, including robust OOD detection with the proposed rODIN method. The findings underscore the value of explicit attribute control in SER and call for task-tailored OOD strategies to better address real-world deployment challenges.

Abstract

Despite notable progress, speech emotion recognition (SER) remains challenging due to the intricate and ambiguous nature of speech emotion, particularly in wild world. While current studies primarily focus on recognition and generalization abilities, our research pioneers an investigation into the reliability of SER methods in the presence of semantic data shifts and explores how to exert fine-grained control over various attributes inherent in speech signals to enhance speech emotion modeling. In this paper, we first introduce MSAC-SERNet, a novel unified SER framework capable of simultaneously handling both single-corpus and cross-corpus SER. Specifically, concentrating exclusively on the speech emotion attribute, a novel CNN-based SER model is presented to extract discriminative emotional representations, guided by additive margin softmax loss. Considering information overlap between various speech attributes, we propose a novel learning paradigm based on correlations of different speech attributes, termed Multiple Speech Attribute Control (MSAC), which empowers the proposed SER model to simultaneously capture fine-grained emotion-related features while mitigating the negative impact of emotion-agnostic representations. Furthermore, we make a first attempt to examine the reliability of the MSAC-SERNet framework using out-of-distribution detection methods. Experiments on both single-corpus and cross-corpus SER scenarios indicate that MSAC-SERNet not only consistently outperforms the baseline in all aspects, but achieves superior performance compared to state-of-the-art SER approaches.

MSAC: Multiple Speech Attribute Control Method for Reliable Speech Emotion Recognition

TL;DR

This work tackles the reliability of speech emotion recognition under semantic data shifts by introducing MSAC-SERNet, a unified framework that jointly handles single-corpus and cross-corpus SER. It introduces Multiple Speech Attribute Control (MSAC), a paradigm that models correlations among speech attributes and mitigates emotion-agnostic representations, aided by FBanks inputs and an AM-Softmax-based loss. The approach achieves state-of-the-art results on six public corpora, demonstrating improved recognition, generalization, and reliability, including robust OOD detection with the proposed rODIN method. The findings underscore the value of explicit attribute control in SER and call for task-tailored OOD strategies to better address real-world deployment challenges.

Abstract

Despite notable progress, speech emotion recognition (SER) remains challenging due to the intricate and ambiguous nature of speech emotion, particularly in wild world. While current studies primarily focus on recognition and generalization abilities, our research pioneers an investigation into the reliability of SER methods in the presence of semantic data shifts and explores how to exert fine-grained control over various attributes inherent in speech signals to enhance speech emotion modeling. In this paper, we first introduce MSAC-SERNet, a novel unified SER framework capable of simultaneously handling both single-corpus and cross-corpus SER. Specifically, concentrating exclusively on the speech emotion attribute, a novel CNN-based SER model is presented to extract discriminative emotional representations, guided by additive margin softmax loss. Considering information overlap between various speech attributes, we propose a novel learning paradigm based on correlations of different speech attributes, termed Multiple Speech Attribute Control (MSAC), which empowers the proposed SER model to simultaneously capture fine-grained emotion-related features while mitigating the negative impact of emotion-agnostic representations. Furthermore, we make a first attempt to examine the reliability of the MSAC-SERNet framework using out-of-distribution detection methods. Experiments on both single-corpus and cross-corpus SER scenarios indicate that MSAC-SERNet not only consistently outperforms the baseline in all aspects, but achieves superior performance compared to state-of-the-art SER approaches.
Paper Structure (26 sections, 5 equations, 5 figures, 9 tables)

This paper contains 26 sections, 5 equations, 5 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Speaker-dependent and Speaker-independent SER
  4. Adversarial Domain Adaptation
  5. Out-of-distribution Detection
  6. Methodology
  7. Input Pipeline
  8. Discriminative Feature Extraction
  9. Aggregation Pooling
  10. Loss Function
  11. Multiple Speech Attribute Control Method
  12. Experimental Databases And Setup
  13. Dtatbases
  14. Implementation Details
  15. Evaluation Metrics
  16. ...and 11 more sections

Figures (5)

  • Figure 1: Overall architecture of the proposed unified SER workflow MSAC-SERNet, including three parts: (1) Input pipeline. (2) The proposed SER mdoel. (3) The proposed MSAC learning paradigm.
  • Figure 2: Schematics of the proposed aggregation pooling module.
  • Figure 3: Overview of the proposed MSAC learning paradigm. FC denotes the fully connected layer, GRL represents the gradient reversal layer.
  • Figure 4: Confusion matrices for the variants of the proposed MSAC-SERNet on IEMOCAP. The diagonal numbers represent the recall rate for each emotion.
  • Figure 5: The t-SNE visualization of data distribution on IEMOCAP, where FRU, FEA, SUR represent the emotions of frustration, fear, and surprise.