Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fine-Grained Quantitative Emotion Editing for Speech Generation

Sho Inoue, Kun Zhou, Shuai Wang, Haizhou Li

TL;DR

This work applies a hierarchical emotion distribution extractor, i.e. Hierarchical ED, that quantifies the intensity of emotions at different levels of granularity to the FastSpeech2 framework, guiding the model to learn emotion intensity at phoneme, word, and utterance levels.

Abstract

It remains a significant challenge how to quantitatively control the expressiveness of speech emotion in speech generation. In this work, we present a novel approach for manipulating the rendering of emotions for speech generation. We propose a hierarchical emotion distribution extractor, i.e. Hierarchical ED, that quantifies the intensity of emotions at different levels of granularity. Support vector machines (SVMs) are employed to rank emotion intensity, resulting in a hierarchical emotional embedding. Hierarchical ED is subsequently integrated into the FastSpeech2 framework, guiding the model to learn emotion intensity at phoneme, word, and utterance levels. During synthesis, users can manually edit the emotional intensity of the generated voices. Both objective and subjective evaluations demonstrate the effectiveness of the proposed network in terms of fine-grained quantitative emotion editing.

Fine-Grained Quantitative Emotion Editing for Speech Generation

TL;DR

This work applies a hierarchical emotion distribution extractor, i.e. Hierarchical ED, that quantifies the intensity of emotions at different levels of granularity to the FastSpeech2 framework, guiding the model to learn emotion intensity at phoneme, word, and utterance levels.

Abstract

It remains a significant challenge how to quantitatively control the expressiveness of speech emotion in speech generation. In this work, we present a novel approach for manipulating the rendering of emotions for speech generation. We propose a hierarchical emotion distribution extractor, i.e. Hierarchical ED, that quantifies the intensity of emotions at different levels of granularity. Support vector machines (SVMs) are employed to rank emotion intensity, resulting in a hierarchical emotional embedding. Hierarchical ED is subsequently integrated into the FastSpeech2 framework, guiding the model to learn emotion intensity at phoneme, word, and utterance levels. During synthesis, users can manually edit the emotional intensity of the generated voices. Both objective and subjective evaluations demonstrate the effectiveness of the proposed network in terms of fine-grained quantitative emotion editing.
Paper Structure (17 sections, 2 figures, 4 tables)

This paper contains 17 sections, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Control of Speech Emotion
  4. Speech Editing
  5. Fine-Grained Quantitative Control
  6. Problem Formulation
  7. Hierarchical Emotion Distribution Extractor
  8. Hierarchical Emotion Intensity Modeling
  9. Emotion Editing
  10. Experiments and Results
  11. Experimental Setup
  12. Model Architecture
  13. Results and Analysis
  14. Speech Quality
  15. Emotion Expressiveness
  16. ...and 2 more sections

Figures (2)

  • Figure 1: (a) Model architecture with hierarchical emotion distribution (HED) mechanism in TTS. During inference, the framework extracts hierarchical ED from input audio ("emotion editing"). Users can manually modify hierarchical ED to control emotion intensities at phoneme, word, and utterance levels; (b) Hierarchical ED extraction workflow for emotion distributions at phoneme, word, and utterance levels.
  • Figure 2: The illustration of prosodic variants with intensity changes. The red background represents the expected negative trend, the blue indicates the expected positive trend, both summarized from the ESD dataset. 'U', 'W', 'P', and 'WP' indicate the controlling segments in Hierarchical ED (our model): Utterance, Word, Phoneme, and Word-and-Phoneme, respectively.