Phone-Level Prosody Modelling with GMM-Based MDN for Diverse and Controllable Speech Synthesis

TL;DR

This work proposes a novel approach that models phone-level prosodies with a GMM-based mixture density network(MDN) and then extends it for multi-speaker TTS using speaker adaptation transforms of Gaussian means and variances and shows that it can clone the prosodies from a reference speech by sampling prosody from the Gaussian components that produce the reference prosodies.

Abstract

Generating natural speech with a diverse and smooth prosody pattern is a challenging task. Although random sampling with phone-level prosody distribution has been investigated to generate different prosody patterns, the diversity of the generated speech is still very limited and far from what can be achieved by humans. This is largely due to the use of uni-modal distribution, such as single Gaussian, in the prior works of phone-level prosody modelling. In this work, we propose a novel approach that models phone-level prosodies with a GMM-based mixture density network(MDN) and then extend it for multi-speaker TTS using speaker adaptation transforms of Gaussian means and variances. Furthermore, we show that we can clone the prosodies from a reference speech by sampling prosodies from the Gaussian components that produce the reference prosodies. Our experiments on LJSpeech and LibriTTS dataset show that the proposed method with GMM-based MDN not only achieves significantly better diversity than using a single Gaussian in both single-speaker and multi-speaker TTS, but also provides better naturalness. The prosody cloning experiments demonstrate that the prosody similarity of the proposed method with GMM-based MDN is comparable to recent proposed fine-grained VAE while the target speaker similarity is better.

Figures (9)

• Figure 1: Prosody cloning pipeline with fine-grained VAE.
• Figure 2: The proposed architectures for single speaker TTS. "SG" represents the stop gradient operation. "OR" selects the extracted "ground-truth" $\mathbf{e}$ in training stage and the sampled $\hat{\mathbf{e}}$ in inference stage. We use red lines for loss calculation and dash lines for sampling.
• Figure 3: The extension of the proposed architecture for multi-speaker TTS. The loss function for the TTS system is the same as in Figure \ref{['overall']}, so it is eliminated in \ref{['ms_overall']} for simplicity.
• Figure 4: Speech Synthesis with cloned phone-level prosodies from a reference speech.
• Figure 5: Log-likelihood curves of the extracted phone-level prosodies with different numbers of Gaussian components on LJSpeech and LibriTTS.