Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EmoFormer: A Text-Independent Speech Emotion Recognition using a Hybrid Transformer-CNN model

Rashedul Hasan, Meher Nigar, Nursadul Mamun, Sayan Paul

TL;DR

Speech emotion recognition remains challenging under text- and speaker-independent conditions. EmoFormer introduces a hybrid CNN-Transformer architecture and evaluates two feature pipelines, MFCCs and x-vectors, on META's EARS dataset across 5–23 emotion sets. The MFCC-based model achieves up to 90% accuracy with five emotions, with performance declining as the emotion count increases (83% for seven, 65% for 23), while x-vectors underperform MFCCs. The work demonstrates the merit of combining local and global feature modeling for robust SER and highlights MFCCs as a strong choice in text-independent scenarios, with future work pointing toward multimodal integration and advanced transformer variants.

Abstract

Speech Emotion Recognition is a crucial area of research in human-computer interaction. While significant work has been done in this field, many state-of-the-art networks struggle to accurately recognize emotions in speech when the data is both speech and speaker-independent. To address this limitation, this study proposes, EmoFormer, a hybrid model combining CNNs (CNNs) with Transformer encoders to capture emotion patterns in speech data for such independent datasets. The EmoFormer network was trained and tested using the Expressive Anechoic Recordings of Speech (EARS) dataset, recently released by META. We experimented with two feature extraction techniques: MFCCs and x-vectors. The model was evaluated on different emotion sets comprising 5, 7, 10, and 23 distinct categories. The results demonstrate that the model achieved its best performance with five emotions, attaining an accuracy of 90%, a precision of 0.92, a recall, and an F1-score of 0.91. However, performance decreased as the number of emotions increased, with an accuracy of 83% for seven emotions compared to 70% for the baseline network. This study highlights the effectiveness of combining CNNs and Transformer-based architectures for emotion recognition from speech, particularly when using MFCC features.

EmoFormer: A Text-Independent Speech Emotion Recognition using a Hybrid Transformer-CNN model

TL;DR

Speech emotion recognition remains challenging under text- and speaker-independent conditions. EmoFormer introduces a hybrid CNN-Transformer architecture and evaluates two feature pipelines, MFCCs and x-vectors, on META's EARS dataset across 5–23 emotion sets. The MFCC-based model achieves up to 90% accuracy with five emotions, with performance declining as the emotion count increases (83% for seven, 65% for 23), while x-vectors underperform MFCCs. The work demonstrates the merit of combining local and global feature modeling for robust SER and highlights MFCCs as a strong choice in text-independent scenarios, with future work pointing toward multimodal integration and advanced transformer variants.

Abstract

Speech Emotion Recognition is a crucial area of research in human-computer interaction. While significant work has been done in this field, many state-of-the-art networks struggle to accurately recognize emotions in speech when the data is both speech and speaker-independent. To address this limitation, this study proposes, EmoFormer, a hybrid model combining CNNs (CNNs) with Transformer encoders to capture emotion patterns in speech data for such independent datasets. The EmoFormer network was trained and tested using the Expressive Anechoic Recordings of Speech (EARS) dataset, recently released by META. We experimented with two feature extraction techniques: MFCCs and x-vectors. The model was evaluated on different emotion sets comprising 5, 7, 10, and 23 distinct categories. The results demonstrate that the model achieved its best performance with five emotions, attaining an accuracy of 90%, a precision of 0.92, a recall, and an F1-score of 0.91. However, performance decreased as the number of emotions increased, with an accuracy of 83% for seven emotions compared to 70% for the baseline network. This study highlights the effectiveness of combining CNNs and Transformer-based architectures for emotion recognition from speech, particularly when using MFCC features.
Paper Structure (11 sections, 7 equations, 3 figures, 4 tables)

This paper contains 11 sections, 7 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Dataset
  4. Data Augmentation
  5. Feature Extraction
  6. Mel-frequency cepstral coefficients
  7. X-vector Feature Extraction
  8. Model Architecture
  9. Experimental Setup
  10. Results
  11. Conclusion

Figures (3)

  • Figure 1: MFCC features for different emotions such as Adortion, Sadness, Anger, and Neutral.
  • Figure 2: Architecture of the proposed EmoFormer network
  • Figure 3: Confusion matrix generated using a) MFCC and b) X-vector