Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ASR for Affective Speech: Investigating Impact of Emotion and Speech Generative Strategy

Ya-Tse Wu, Chi-Chun Lee

TL;DR

Emotion in speech can degrade ASR performance, especially through substitutions that disrupt phonetic realization. The paper uses three emotion-controllable TTS models to synthesize labeled data, analyzes transcription correctness and emotional salience, and develops two data-selection strategies plus a combined approach to guide fine-tuning. The TTS-G, EMO-G, and especially the combined TTS-EMO-G strategy improve WER on real emotional datasets while preserving performance on neutral speech, with MaskGCT-based synthetic data providing the strongest gains. These results demonstrate that targeted augmentation leveraging emotionally expressive yet linguistically reliable samples is a practical path to building robust, emotion-aware ASR systems.

Abstract

This work investigates how emotional speech and generative strategies affect ASR performance. We analyze speech synthesized from three emotional TTS models and find that substitution errors dominate, with emotional expressiveness varying across models. Based on these insights, we introduce two generative strategies: one using transcription correctness and another using emotional salience, to construct fine-tuning subsets. Results show consistent WER improvements on real emotional datasets without noticeable degradation on clean LibriSpeech utterances. The combined strategy achieves the strongest gains, particularly for expressive speech. These findings highlight the importance of targeted augmentation for building emotion-aware ASR systems.

ASR for Affective Speech: Investigating Impact of Emotion and Speech Generative Strategy

TL;DR

Emotion in speech can degrade ASR performance, especially through substitutions that disrupt phonetic realization. The paper uses three emotion-controllable TTS models to synthesize labeled data, analyzes transcription correctness and emotional salience, and develops two data-selection strategies plus a combined approach to guide fine-tuning. The TTS-G, EMO-G, and especially the combined TTS-EMO-G strategy improve WER on real emotional datasets while preserving performance on neutral speech, with MaskGCT-based synthetic data providing the strongest gains. These results demonstrate that targeted augmentation leveraging emotionally expressive yet linguistically reliable samples is a practical path to building robust, emotion-aware ASR systems.

Abstract

This work investigates how emotional speech and generative strategies affect ASR performance. We analyze speech synthesized from three emotional TTS models and find that substitution errors dominate, with emotional expressiveness varying across models. Based on these insights, we introduce two generative strategies: one using transcription correctness and another using emotional salience, to construct fine-tuning subsets. Results show consistent WER improvements on real emotional datasets without noticeable degradation on clean LibriSpeech utterances. The combined strategy achieves the strongest gains, particularly for expressive speech. These findings highlight the importance of targeted augmentation for building emotion-aware ASR systems.
Paper Structure (23 sections, 2 equations, 4 figures, 5 tables)

This paper contains 23 sections, 2 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Synthesized Data Analysis and Emotion Impact on ASR System
  3. Emotion Controllable TTS Data Synthesis
  4. CosyVoice2 du2024cosyvoice
  5. EmoVoice yang2025emovoice
  6. MaskGCT wang2025maskgct
  7. Synthesized Data Analysis Methods
  8. TTS Correctness Analysis
  9. Emotion Salience Analysis
  10. Result and Discussion
  11. Generative Strategies and Comparison on Synthesized Data
  12. Generative Strategy
  13. TTS Correctness (TTS-G)
  14. Emotion Salience (EMO-G)
  15. Combined Generative Strategy (TTS-EMO-G)
  16. ...and 8 more sections

Figures (4)

  • Figure 1: Workflow for synthesized emotional speech analysis.
  • Figure 2: Distribution of word error types in the training set of emotional TTS speech.
  • Figure 3: Box plots of emotion distributions in the training set of emotional TTS speech.
  • Figure 4: Left to right: (Act, Val), (Val, Dom), and (Act, Dom) planes showing WER difference patterns for MSP-Test 2 (top) and IEMOCAP (bottom).