Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-Channel Speech Enhancement for Cocktail Party Speech Emotion Recognition

Youjun Chen, Guinan Li, Mengzhe Geng, Xurong Xie, Shujie Hu, Huimeng Wang, Haoning Xu, Chengxi Deng, Jiajun Deng, Zhaoqing Li, Mingyu Cui, Xunying Liu

TL;DR

Experiments on mixture speech constructed using the IEMOCAP and MSP-FACE datasets suggest the MCSE output consistently outperforms domain fine-tuned single-channel speech representations produced by: a) Conformer-based metric GANs; and b) WavLM SSL features with optional SE-ER dual task fine-tuning.

Abstract

This paper highlights the critical importance of multi-channel speech enhancement (MCSE) for speech emotion recognition (ER) in cocktail party scenarios. A multi-channel speech dereverberation and separation front-end integrating DNN-WPE and mask-based MVDR is used to extract the target speaker's speech from the mixture speech, before being fed into the downstream ER back-end using HuBERT- and ViT-based speech and visual features. Experiments on mixture speech constructed using the IEMOCAP and MSP-FACE datasets suggest the MCSE output consistently outperforms domain fine-tuned single-channel speech representations produced by: a) Conformer-based metric GANs; and b) WavLM SSL features with optional SE-ER dual task fine-tuning. Statistically significant increases in weighted, unweighted accuracy and F1 measures by up to 9.5%, 8.5% and 9.1% absolute (17.1%, 14.7% and 16.0% relative) are obtained over the above single-channel baselines. The generalization of IEMOCAP trained MCSE front-ends are also shown when being zero-shot applied to out-of-domain MSP-FACE data.

Multi-Channel Speech Enhancement for Cocktail Party Speech Emotion Recognition

TL;DR

Experiments on mixture speech constructed using the IEMOCAP and MSP-FACE datasets suggest the MCSE output consistently outperforms domain fine-tuned single-channel speech representations produced by: a) Conformer-based metric GANs; and b) WavLM SSL features with optional SE-ER dual task fine-tuning.

Abstract

This paper highlights the critical importance of multi-channel speech enhancement (MCSE) for speech emotion recognition (ER) in cocktail party scenarios. A multi-channel speech dereverberation and separation front-end integrating DNN-WPE and mask-based MVDR is used to extract the target speaker's speech from the mixture speech, before being fed into the downstream ER back-end using HuBERT- and ViT-based speech and visual features. Experiments on mixture speech constructed using the IEMOCAP and MSP-FACE datasets suggest the MCSE output consistently outperforms domain fine-tuned single-channel speech representations produced by: a) Conformer-based metric GANs; and b) WavLM SSL features with optional SE-ER dual task fine-tuning. Statistically significant increases in weighted, unweighted accuracy and F1 measures by up to 9.5%, 8.5% and 9.1% absolute (17.1%, 14.7% and 16.0% relative) are obtained over the above single-channel baselines. The generalization of IEMOCAP trained MCSE front-ends are also shown when being zero-shot applied to out-of-domain MSP-FACE data.
Paper Structure (15 sections, 5 equations, 1 figure, 2 tables)

This paper contains 15 sections, 5 equations, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Single-Channel Speech Enhancement
  3. Multi-Channel Speech Enhancement
  4. DNN-WPE for Speech Dereverberation
  5. Mask-based MVDR for Speech Separation
  6. Emotion Recognition Back-end
  7. Audio Encoder
  8. Visual Encoder
  9. Three Different Emotion Decoders
  10. Experiments
  11. Task Description
  12. Experimental Setup
  13. Experimental Results
  14. Conclusion
  15. Acknowledgements

Figures (1)

  • Figure 1: Illustration of multi-channel speech enhancement (MCSE) and emotion recognition (ER) system. The MCSE front-end integrates DNN-WPE based speech dereverberation and mask-based MVDR speech separation components. Three different emotion recognition decoders are designed for audio-only and audio-visual ER systems, including: (a) audio-only ER system (MCSE-ER (Audio-only)), (b) early-fusion audio-visual ER system (MCSE-ER (AV-Early)) using cross-modal attention-based fusion block and (c) late-fusion audio-visual ER system (MCSE-ER (AV-Late)) using the weights of audio ($w_{a}$) and visual ($w_{v}$) to calculate the final classified probability.