Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MultiVerse: Efficient and Expressive Zero-Shot Multi-Task Text-to-Speech

Taejun Bak, Youngsik Eom, SeungJae Choi, Young-Sun Joo

TL;DR

A novel prosody modeling technique significantly contributes to MultiVerse's ability to generate speech with high prosody similarity to the given prompts, and significantly outperforms other zero-shot TTS systems trained with the same small amount of data.

Abstract

Text-to-speech (TTS) systems that scale up the amount of training data have achieved significant improvements in zero-shot speech synthesis. However, these systems have certain limitations: they require a large amount of training data, which increases costs, and often overlook prosody similarity. To address these issues, we propose MultiVerse, a zero-shot multi-task TTS system that is able to perform TTS or speech style transfer in zero-shot and cross-lingual conditions. MultiVerse requires much less training data than traditional data-driven approaches. To ensure zero-shot performance even with limited data, we leverage source-filter theory-based disentanglement, utilizing the prompt for modeling filter-related and source-related representations. Additionally, to further enhance prosody similarity, we adopt a prosody modeling approach combining prompt-based autoregressive and non-autoregressive methods. Evaluations demonstrate the remarkable zero-shot multi-task TTS performance of MultiVerse and show that MultiVerse not only achieves zero-shot TTS performance comparable to data-driven TTS systems with much less data, but also significantly outperforms other zero-shot TTS systems trained with the same small amount of data. In particular, our novel prosody modeling technique significantly contributes to MultiVerse's ability to generate speech with high prosody similarity to the given prompts. Our samples are available at https://nc-ai.github.io/speech/publications/multiverse/index.html

MultiVerse: Efficient and Expressive Zero-Shot Multi-Task Text-to-Speech

TL;DR

A novel prosody modeling technique significantly contributes to MultiVerse's ability to generate speech with high prosody similarity to the given prompts, and significantly outperforms other zero-shot TTS systems trained with the same small amount of data.

Abstract

Text-to-speech (TTS) systems that scale up the amount of training data have achieved significant improvements in zero-shot speech synthesis. However, these systems have certain limitations: they require a large amount of training data, which increases costs, and often overlook prosody similarity. To address these issues, we propose MultiVerse, a zero-shot multi-task TTS system that is able to perform TTS or speech style transfer in zero-shot and cross-lingual conditions. MultiVerse requires much less training data than traditional data-driven approaches. To ensure zero-shot performance even with limited data, we leverage source-filter theory-based disentanglement, utilizing the prompt for modeling filter-related and source-related representations. Additionally, to further enhance prosody similarity, we adopt a prosody modeling approach combining prompt-based autoregressive and non-autoregressive methods. Evaluations demonstrate the remarkable zero-shot multi-task TTS performance of MultiVerse and show that MultiVerse not only achieves zero-shot TTS performance comparable to data-driven TTS systems with much less data, but also significantly outperforms other zero-shot TTS systems trained with the same small amount of data. In particular, our novel prosody modeling technique significantly contributes to MultiVerse's ability to generate speech with high prosody similarity to the given prompts. Our samples are available at https://nc-ai.github.io/speech/publications/multiverse/index.html
Paper Structure (31 sections, 1 equation, 9 figures, 8 tables)

This paper contains 31 sections, 1 equation, 9 figures, 8 tables.

Table of Contents

  1. Introduction
  2. MultiVerse
  3. Source-Filter Theory Based Decomposed Modeling
  4. Increasing Filter Capacity of the Acoustic Decoder
  5. Two-stage Prosody Modeling
  6. Autoregressive Prosody Modeling
  7. Non-Autoregressive Prosody Modeling
  8. Learning Objectives
  9. Multi-Task TTS
  10. Experimental Environments
  11. Datasets
  12. Experimental Setup
  13. Experimental Results
  14. Zero-shot Scenario
  15. Comparison with Data-Driven Models
  16. ...and 16 more sections

Figures (9)

  • Figure 1: Overall structure of MultiVerse. The acoustic model and the autoregressive prosody predictor are on the left and right side of the figure, respectively. During training, overall modules are trained together, except the pre-trained speaker encoder. Multi-task TTS can be accomplished by varying input conditions.
  • Figure 2: Style transfer process to transfer speech style from speaker B to speaker A. The acoustic decoder is omitted for simplicity.
  • Figure 3: Violin plot describing duration and pitch distributions.
  • Figure 4: $F_0$ contours of pitch-shifted synthesized speech whose predicted pitch units are manipulated with $\{-6, -4, -2, 2, 4, 6\}$.
  • Figure 5: Mel-spectrograms of synthesized audio sample: (a) generated from the filter representation only. (b) generated from the source representation only. (c) generated from the coarse mel-representation.
  • ...and 4 more figures