Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EmoSphere-TTS: Emotional Style and Intensity Modeling via Spherical Emotion Vector for Controllable Emotional Text-to-Speech

Deok-Hyeon Cho, Hyung-Seok Oh, Seung-Bin Kim, Sang-Hoon Lee, Seong-Whan Lee

TL;DR

EmoSphere-TTS is proposed, which synthesizes expressive emotional speech by using a spherical emotion vector to control the emotional style and intensity of the synthetic speech and a dual conditional adversarial network to improve the quality of generated speech by reflecting the multi-aspect characteristics.

Abstract

Despite rapid advances in the field of emotional text-to-speech (TTS), recent studies primarily focus on mimicking the average style of a particular emotion. As a result, the ability to manipulate speech emotion remains constrained to several predefined labels, compromising the ability to reflect the nuanced variations of emotion. In this paper, we propose EmoSphere-TTS, which synthesizes expressive emotional speech by using a spherical emotion vector to control the emotional style and intensity of the synthetic speech. Without any human annotation, we use the arousal, valence, and dominance pseudo-labels to model the complex nature of emotion via a Cartesian-spherical transformation. Furthermore, we propose a dual conditional adversarial network to improve the quality of generated speech by reflecting the multi-aspect characteristics. The experimental results demonstrate the model ability to control emotional style and intensity with high-quality expressive speech.

EmoSphere-TTS: Emotional Style and Intensity Modeling via Spherical Emotion Vector for Controllable Emotional Text-to-Speech

TL;DR

EmoSphere-TTS is proposed, which synthesizes expressive emotional speech by using a spherical emotion vector to control the emotional style and intensity of the synthetic speech and a dual conditional adversarial network to improve the quality of generated speech by reflecting the multi-aspect characteristics.

Abstract

Despite rapid advances in the field of emotional text-to-speech (TTS), recent studies primarily focus on mimicking the average style of a particular emotion. As a result, the ability to manipulate speech emotion remains constrained to several predefined labels, compromising the ability to reflect the nuanced variations of emotion. In this paper, we propose EmoSphere-TTS, which synthesizes expressive emotional speech by using a spherical emotion vector to control the emotional style and intensity of the synthetic speech. Without any human annotation, we use the arousal, valence, and dominance pseudo-labels to model the complex nature of emotion via a Cartesian-spherical transformation. Furthermore, we propose a dual conditional adversarial network to improve the quality of generated speech by reflecting the multi-aspect characteristics. The experimental results demonstrate the model ability to control emotional style and intensity with high-quality expressive speech.
Paper Structure (16 sections, 6 equations, 3 figures, 2 tables)

This paper contains 16 sections, 6 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. EmoSphere-TTS
  3. Emotional style and intensity modeling
  4. AVD encoder
  5. Cartesian-spherical transformation
  6. Spherical emotion encoder
  7. Dual conditional adversarial training
  8. TTS model
  9. Experiments and results
  10. Experimental setup
  11. Implementation details
  12. Model performance
  13. Emotion intensity controllability
  14. Emotion style shift
  15. Conclusion
  16. ...and 1 more sections

Figures (3)

  • Figure 1: Overall architecture of EmoSphere-TTS.
  • Figure 2: The pitch tendency according to emotion and intensity.
  • Figure 3: A pitch track of a sample demonstrating the effects of emotional style shift, where $A$, $V$, and $D$ represent arousal, valence, and dominance, respectively. The line color represents emotional intensity, red = 0.1, green = 0.5, and blue = 0.9.