Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring Multilingual Unseen Speaker Emotion Recognition: Leveraging Co-Attention Cues in Multitask Learning

Arnav Goel, Medha Hira, Anubha Gupta

TL;DR

The paper tackles the challenge of unseen-speaker multilingual speech emotion recognition by introducing CAMuLeNet, a co-attention fusion network that combines frequency-domain features with Whisper-based embeddings within a multitask learning framework for emotion and gender. It systematically benchmarks pretrained encoders across six multilingual datasets, including the novel BhavVani Hindi SER dataset, using a rigorous 10-fold leave-speaker-out evaluation, and reports an average improvement of approximately 8% on unseen speakers. The work contributes BhavVani as the first open-source Hindi SER dataset, provides a comprehensive benchmark of multiple PTMs on multilingual SER, and demonstrates that co-attention fusion plus multitask learning yields robust improvements, with CREMA-D and RAVDESS showing notable gains. These results advance robust, multilingual SER and have practical implications for deploying emotion recognition in diverse real-world scenarios.

Abstract

Advent of modern deep learning techniques has given rise to advancements in the field of Speech Emotion Recognition (SER). However, most systems prevalent in the field fail to generalize to speakers not seen during training. This study focuses on handling challenges of multilingual SER, specifically on unseen speakers. We introduce CAMuLeNet, a novel architecture leveraging co-attention based fusion and multitask learning to address this problem. Additionally, we benchmark pretrained encoders of Whisper, HuBERT, Wav2Vec2.0, and WavLM using 10-fold leave-speaker-out cross-validation on five existing multilingual benchmark datasets: IEMOCAP, RAVDESS, CREMA-D, EmoDB and CaFE and, release a novel dataset for SER on the Hindi language (BhavVani). CAMuLeNet shows an average improvement of approximately 8% over all benchmarks on unseen speakers determined by our cross-validation strategy.

Exploring Multilingual Unseen Speaker Emotion Recognition: Leveraging Co-Attention Cues in Multitask Learning

TL;DR

The paper tackles the challenge of unseen-speaker multilingual speech emotion recognition by introducing CAMuLeNet, a co-attention fusion network that combines frequency-domain features with Whisper-based embeddings within a multitask learning framework for emotion and gender. It systematically benchmarks pretrained encoders across six multilingual datasets, including the novel BhavVani Hindi SER dataset, using a rigorous 10-fold leave-speaker-out evaluation, and reports an average improvement of approximately 8% on unseen speakers. The work contributes BhavVani as the first open-source Hindi SER dataset, provides a comprehensive benchmark of multiple PTMs on multilingual SER, and demonstrates that co-attention fusion plus multitask learning yields robust improvements, with CREMA-D and RAVDESS showing notable gains. These results advance robust, multilingual SER and have practical implications for deploying emotion recognition in diverse real-world scenarios.

Abstract

Advent of modern deep learning techniques has given rise to advancements in the field of Speech Emotion Recognition (SER). However, most systems prevalent in the field fail to generalize to speakers not seen during training. This study focuses on handling challenges of multilingual SER, specifically on unseen speakers. We introduce CAMuLeNet, a novel architecture leveraging co-attention based fusion and multitask learning to address this problem. Additionally, we benchmark pretrained encoders of Whisper, HuBERT, Wav2Vec2.0, and WavLM using 10-fold leave-speaker-out cross-validation on five existing multilingual benchmark datasets: IEMOCAP, RAVDESS, CREMA-D, EmoDB and CaFE and, release a novel dataset for SER on the Hindi language (BhavVani). CAMuLeNet shows an average improvement of approximately 8% over all benchmarks on unseen speakers determined by our cross-validation strategy.
Paper Structure (22 sections, 4 equations, 3 figures, 1 table)

This paper contains 22 sections, 4 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Proposed Methodology
  3. Extracting Features from the Frequency Domain
  4. Extracting Features through Transfer Learning
  5. Co-Attention based Fusion
  6. Multitask Learning
  7. Datasets
  8. Benchmark Datasets
  9. Our Novel Hindi SER Dataset: BhavVani
  10. Data Collection
  11. Data Annotation and Validation
  12. Dataset Statistics
  13. Experiments
  14. Baseline Pre-Trained Models Employed
  15. Baseline Experiment Setup
  16. ...and 7 more sections

Figures (3)

  • Figure 1: CAMuLeNet: Co-Attention based Multitask Learning Network
  • Figure 2: BhavVani Dataset Statistics
  • Figure 3: t-SNE visualisation of Feature Distribution. (a) Distribution of final extracted features from Whisper-Medium on the entire CREMA-D dataset. (b) Distribution of final extracted features from CAMuLeNet before the final classification layer on the entire CREMA-D dataset.