Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron

TL;DR

The paper tackles end-to-end prosody transfer for expressive TTS by learning a fixed-length prosody embedding from a reference audio and conditioning Tacotron on it. The reference encoder enables cross-speaker prosody transfer and even text-dependent templated prosody, validated with objective metrics and human judgments on single and 44-speaker setups. However, prosody transfer can entangle pitch with speaker identity, revealing a need for pitch-relative representations and better disentanglement of text, prosody, and voice. The work also defines evaluation metrics and outlines future directions, including sampling from the learned prosody space and learning priors.

Abstract

We present an extension to the Tacotron speech synthesis architecture that learns a latent embedding space of prosody, derived from a reference acoustic representation containing the desired prosody. We show that conditioning Tacotron on this learned embedding space results in synthesized audio that matches the prosody of the reference signal with fine time detail even when the reference and synthesis speakers are different. Additionally, we show that a reference prosody embedding can be used to synthesize text that is different from that of the reference utterance. We define several quantitative and subjective metrics for evaluating prosody transfer, and report results with accompanying audio samples from single-speaker and 44-speaker Tacotron models on a prosody transfer task.

Figures (7)

• Figure 1: The full Tacotron architecture for prosody control. The autoregressive decoder is conditioned on the result of the reference encoder, transcript encoder, and speaker embedding via an attention module.
• Figure 2: The prosody reference encoder module. A 6-layer stack of 2D convolutions with batch normalization, followed by "recurrent pooling" to summarize the variable length sequence, followed by an optional fully connected layer and activation.
• Figure 3: An interpretation of the Tacotron architecture for prosody control from Figure \ref{['fig:tacotron-conditioning']} as an RNN encoder-decoder with speaker and phonetic conditioning input.
• Figure 4: Mel spectrograms for the utterance "Snuffles is a lot happier. And smells a lot better." (Top) Reference utterance from an unseen speaker. (Middle) Synthesized utterance conditioned on reference embedding. (Bottom) Synthesized utterance from a model without reference conditioning.
• Figure 5: Pitch tracks for the utterance "Snuffles is a lot happier. And smells a lot better." A pitch of 0 Hz indicates an unvoiced segment. (Top) Reference utterance from an unseen speaker. (Middle) Synthesized utterance conditioned on reference embedding. (Bottom) Synthesized utterance from a model without reference conditioning.

Theorems & Definitions (1)

• Definition