Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Realistic Emotional Voice Conversion using Controllable Emotional Intensity

Tianhua Qi, Shiyan Wang, Cheng Lu, Yan Zhao, Yuan Zong, Wenming Zheng

TL;DR

This work tackles the challenge of limited emotional diversity in emotional voice conversion by introducing EINet, a controllable, intensity-aware CVAE framework. EINet combines a posterior network, prior network, intensity mapper, emotion evaluator, emotion renderer, and duration predictor to model valence-arousal-dominance (VAD) dynamics and rhythm changes, using intensity pseudo-labels derived from VAD values via a reversible normalizing flow. The approach yields improved objective metrics (e.g., lower RMSE_F0 and DDUR) and higher MOS scores compared to strong baselines, while ablations confirm the importance of each component for naturalness, controllability, and diversity. The results demonstrate that explicit, VAD-guided intensity control can produce more realistic and expressive emotional voices, with potential for future text-based emotion editing to further enhance controllability and applicability in HCI systems.

Abstract

Realistic emotional voice conversion (EVC) aims to enhance emotional diversity of converted audios, making the synthesized voices more authentic and natural. To this end, we propose Emotional Intensity-aware Network (EINet), dynamically adjusting intonation and rhythm by incorporating controllable emotional intensity. To better capture nuances in emotional intensity, we go beyond mere distance measurements among acoustic features. Instead, an emotion evaluator is utilized to precisely quantify speaker's emotional state. By employing an intensity mapper, intensity pseudo-labels are obtained to bridge the gap between emotional speech intensity modeling and run-time conversion. To ensure high speech quality while retaining controllability, an emotion renderer is used for combining linguistic features smoothly with manipulated emotional features at frame level. Furthermore, we employ a duration predictor to facilitate adaptive prediction of rhythm changes condition on specifying intensity value. Experimental results show EINet's superior performance in naturalness and diversity of emotional expression compared to state-of-the-art EVC methods.

Towards Realistic Emotional Voice Conversion using Controllable Emotional Intensity

TL;DR

This work tackles the challenge of limited emotional diversity in emotional voice conversion by introducing EINet, a controllable, intensity-aware CVAE framework. EINet combines a posterior network, prior network, intensity mapper, emotion evaluator, emotion renderer, and duration predictor to model valence-arousal-dominance (VAD) dynamics and rhythm changes, using intensity pseudo-labels derived from VAD values via a reversible normalizing flow. The approach yields improved objective metrics (e.g., lower RMSE_F0 and DDUR) and higher MOS scores compared to strong baselines, while ablations confirm the importance of each component for naturalness, controllability, and diversity. The results demonstrate that explicit, VAD-guided intensity control can produce more realistic and expressive emotional voices, with potential for future text-based emotion editing to further enhance controllability and applicability in HCI systems.

Abstract

Realistic emotional voice conversion (EVC) aims to enhance emotional diversity of converted audios, making the synthesized voices more authentic and natural. To this end, we propose Emotional Intensity-aware Network (EINet), dynamically adjusting intonation and rhythm by incorporating controllable emotional intensity. To better capture nuances in emotional intensity, we go beyond mere distance measurements among acoustic features. Instead, an emotion evaluator is utilized to precisely quantify speaker's emotional state. By employing an intensity mapper, intensity pseudo-labels are obtained to bridge the gap between emotional speech intensity modeling and run-time conversion. To ensure high speech quality while retaining controllability, an emotion renderer is used for combining linguistic features smoothly with manipulated emotional features at frame level. Furthermore, we employ a duration predictor to facilitate adaptive prediction of rhythm changes condition on specifying intensity value. Experimental results show EINet's superior performance in naturalness and diversity of emotional expression compared to state-of-the-art EVC methods.
Paper Structure (14 sections, 12 equations, 3 figures, 3 tables)

This paper contains 14 sections, 12 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Proposed Method
  3. Posterior network
  4. Intensity mapper
  5. Prior network
  6. Final loss
  7. Experiments
  8. Experimental setup
  9. Model performance
  10. Ablation study
  11. Controllability of emotional intensity
  12. Diversity of emotional samples
  13. Conclusion
  14. Acknowledgements

Figures (3)

  • Figure 1: Diagram of our proposed EINet, depicting the training procedure(a) and inference procedure(b).
  • Figure 2: Pitch and energy tracks of a testing clip.
  • Figure 3: Mel-spectrograms and F0 contours of converted audios at different emotional intensity.