Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Speech Emotion Recognition with ASR Integration

Yuanchao Li

TL;DR

This thesis addresses the challenge of robust Speech Emotion Recognition (SER) in real-world, spontaneous speech by integrating Automatic Speech Recognition (ASR) into SER workflows. It advances understanding of how modern speech foundation models encode emotion, demonstrates how ASR-derived transcripts and hidden representations can augment SER, and develops robust, multi-modal fusion strategies that mitigate ASR errors and cross-modal incongruity. It also proposes practical avenues to scale SER through semi-supervised learning and demonstrates extensions to Alzheimer's dementia detection, underscoring broad applicability. Collectively, the work offers concrete methods for leveraging ASR and self-supervised representations to build more interpretable, scalable, and reliable emotion-aware systems with real-world impact. The findings highlight the nuanced trade-offs between acoustic and lexical cues, the value of intermediate fusion and dynamic modality gating, and the promise of error-aware approaches and LLM/S2S-based error correction to maintain SER performance under imperfect transcripts.

Abstract

Speech Emotion Recognition (SER) plays a pivotal role in understanding human communication, enabling emotionally intelligent systems, and serving as a fundamental component in the development of Artificial General Intelligence (AGI). However, deploying SER in real-world, spontaneous, and low-resource scenarios remains a significant challenge due to the complexity of emotional expression and the limitations of current speech and language technologies. This thesis investigates the integration of Automatic Speech Recognition (ASR) into SER, with the goal of enhancing the robustness, scalability, and practical applicability of emotion recognition from spoken language.

Speech Emotion Recognition with ASR Integration

TL;DR

This thesis addresses the challenge of robust Speech Emotion Recognition (SER) in real-world, spontaneous speech by integrating Automatic Speech Recognition (ASR) into SER workflows. It advances understanding of how modern speech foundation models encode emotion, demonstrates how ASR-derived transcripts and hidden representations can augment SER, and develops robust, multi-modal fusion strategies that mitigate ASR errors and cross-modal incongruity. It also proposes practical avenues to scale SER through semi-supervised learning and demonstrates extensions to Alzheimer's dementia detection, underscoring broad applicability. Collectively, the work offers concrete methods for leveraging ASR and self-supervised representations to build more interpretable, scalable, and reliable emotion-aware systems with real-world impact. The findings highlight the nuanced trade-offs between acoustic and lexical cues, the value of intermediate fusion and dynamic modality gating, and the promise of error-aware approaches and LLM/S2S-based error correction to maintain SER performance under imperfect transcripts.

Abstract

Speech Emotion Recognition (SER) plays a pivotal role in understanding human communication, enabling emotionally intelligent systems, and serving as a fundamental component in the development of Artificial General Intelligence (AGI). However, deploying SER in real-world, spontaneous, and low-resource scenarios remains a significant challenge due to the complexity of emotional expression and the limitations of current speech and language technologies. This thesis investigates the integration of Automatic Speech Recognition (ASR) into SER, with the goal of enhancing the robustness, scalability, and practical applicability of emotion recognition from spoken language.
Paper Structure (172 sections, 30 equations, 28 figures, 46 tables, 3 algorithms)

This paper contains 172 sections, 30 equations, 28 figures, 46 tables, 3 algorithms.

Table of Contents

  1. Lay Summary
  2. Acknowledgements
  3. Declaration
  4. Introduction
  5. Motivation
  6. Research Questions and Objectives
  7. Thesis Outline and Contributions
  8. Overview of SER and Background of ASR Integration
  9. Introduction
  10. Emotion Theories
  11. Categorical Emotion Theory
  12. Dimensional Emotion Theory
  13. Alternative Theories
  14. SER as A Machine Learning Task
  15. Feature Extraction
  16. ...and 157 more sections

Figures (28)

  • Figure 1: Wav2vec 2.0 model.
  • Figure 2: SER accuracy comparison using models. PT: pre-trained; FT100: fine-tuned on 100h of Librispeech; FT960: fine-tuned on 960h of Librispeech.
  • Figure 3: Pair-wise correlations of layer representations.
  • Figure 4: Hierarchical CCA similarity differences.
  • Figure 5: Discriminative analysis for emotion bias.
  • ...and 23 more figures