Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EmoTech: A Multi-modal Speech Emotion Recognition Using Multi-source Low-level Information with Hybrid Recurrent Network

Shamin Bin Habib Avro, Taieba Taher, Nursadul Mamun

TL;DR

EmoTech addresses the challenge of accurate speech emotion recognition by fusing low-level audio and text features in a unified multimodal framework. It employs parallel Audio and Text Blocks that leverage BiLSTM and CNN architectures to extract temporal and local features, respectively, before concatenating them for final classification. The model, evaluated on the IEMOCAP dataset with data augmentation and five emotion classes, achieves a substantial improvement over existing single-modality and some multimodal approaches, reaching about 83.5% accuracy. This approach enhances robustness to class imbalance and demonstrates practical potential for improved human-computer interaction, with future work extending to video modalities.

Abstract

Emotion recognition is a critical task in human-computer interaction, enabling more intuitive and responsive systems. This study presents a multimodal emotion recognition system that combines low-level information from audio and text, leveraging both Convolutional Neural Networks (CNNs) and Bidirectional Long Short-Term Memory Networks (BiLSTMs). The proposed system consists of two parallel networks: an Audio Block and a Text Block. Mel Frequency Cepstral Coefficients (MFCCs) are extracted and processed by a BiLSTM network and a 2D convolutional network to capture low-level intrinsic and extrinsic features from speech. Simultaneously, a combined BiLSTM-CNN network extracts the low-level sequential nature of text from word embeddings corresponding to the available audio. This low-level information from speech and text is then concatenated and processed by several fully connected layers to classify the speech emotion. Experimental results demonstrate that the proposed EmoTech accurately recognizes emotions from combined audio and text inputs, achieving an overall accuracy of 84%. This solution outperforms previously proposed approaches for the same dataset and modalities.

EmoTech: A Multi-modal Speech Emotion Recognition Using Multi-source Low-level Information with Hybrid Recurrent Network

TL;DR

EmoTech addresses the challenge of accurate speech emotion recognition by fusing low-level audio and text features in a unified multimodal framework. It employs parallel Audio and Text Blocks that leverage BiLSTM and CNN architectures to extract temporal and local features, respectively, before concatenating them for final classification. The model, evaluated on the IEMOCAP dataset with data augmentation and five emotion classes, achieves a substantial improvement over existing single-modality and some multimodal approaches, reaching about 83.5% accuracy. This approach enhances robustness to class imbalance and demonstrates practical potential for improved human-computer interaction, with future work extending to video modalities.

Abstract

Emotion recognition is a critical task in human-computer interaction, enabling more intuitive and responsive systems. This study presents a multimodal emotion recognition system that combines low-level information from audio and text, leveraging both Convolutional Neural Networks (CNNs) and Bidirectional Long Short-Term Memory Networks (BiLSTMs). The proposed system consists of two parallel networks: an Audio Block and a Text Block. Mel Frequency Cepstral Coefficients (MFCCs) are extracted and processed by a BiLSTM network and a 2D convolutional network to capture low-level intrinsic and extrinsic features from speech. Simultaneously, a combined BiLSTM-CNN network extracts the low-level sequential nature of text from word embeddings corresponding to the available audio. This low-level information from speech and text is then concatenated and processed by several fully connected layers to classify the speech emotion. Experimental results demonstrate that the proposed EmoTech accurately recognizes emotions from combined audio and text inputs, achieving an overall accuracy of 84%. This solution outperforms previously proposed approaches for the same dataset and modalities.
Paper Structure (13 sections, 7 figures, 3 tables)

This paper contains 13 sections, 7 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Dataset
  4. Network Architecture
  5. Audio Block
  6. Text Block
  7. Classification Block
  8. Hyperparameters and Model Training
  9. Results and Discussions
  10. Effect of data augmentation on EmoTech
  11. Performance evaluation for different classes
  12. Comparison with existing networks
  13. Conclusion

Figures (7)

  • Figure 1: The statistics of ten classes in IEMOCAP Dataset before augmentation
  • Figure 2: The statistics of five dominated classes in IEMOCAP Dataset after augmentation
  • Figure 3: Basic block diagram of the proposed EmoTech Architecture
  • Figure 4: A detail block diagram of audio signal processing in EmoTech
  • Figure 5: A detail block diagram of text data processing in EmoTech
  • ...and 2 more figures