Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ControlSpeech: Towards Simultaneous and Independent Zero-shot Speaker Cloning and Zero-shot Language Style Control

Shengpeng Ji, Qian Chen, Wen Wang, Jialong Zuo, Minghui Fang, Ziyue Jiang, Hai Huang, Zehan Wang, Xize Cheng, Siqi Zheng, Zhou Zhao

TL;DR

<3-5 sentence high-level summary> ControlSpeech introduces a first-of-its-kind TTS framework that simultaneously and independently achieves zero-shot timbre cloning and zero-shot style control by disentangling content, timbre, and style in a discrete codec space and guided by a Style Mixture Semantic Density module. The approach leverages a three-encoder architecture, a mask-based parallel codec generator, and a cross-attention fusion strategy, with SMSD modeling style descriptions as a Gaussian mixture to handle many-to-many mappings and to promote style diversity. Evaluations on the VccmDataset show competitive or state-of-the-art performance across controllability, timbre fidelity, and robustness, including out-of-domain and many-to-many scenarios, and a new dataset is released to support future research. The work also clarifies the limitations of prior approaches and provides open-source code and data to foster community development in controllable TTS.

Abstract

In this paper, we present ControlSpeech, a text-to-speech (TTS) system capable of fully cloning the speaker's voice and enabling arbitrary control and adjustment of speaking style. Prior zero-shot TTS models only mimic the speaker's voice without further control and adjustment capabilities while prior controllable TTS models cannot perform speaker-specific voice generation. Therefore, ControlSpeech focuses on a more challenging task: a TTS system with controllable timbre, content, and style at the same time. ControlSpeech takes speech prompts, content prompts, and style prompts as inputs and utilizes bidirectional attention and mask-based parallel decoding to capture codec representations corresponding to timbre, content, and style in a discrete decoupling codec space. Moreover, we analyze the many-to-many issue in textual style control and propose the Style Mixture Semantic Density (SMSD) module, which is based on Gaussian mixture density networks, to resolve this problem. To facilitate empirical validations, we make available a new style controllable dataset called VccmDataset. Our experimental results demonstrate that ControlSpeech exhibits comparable or state-of-the-art (SOTA) performance in terms of controllability, timbre similarity, audio quality, robustness, and generalizability. The relevant code and demo are available at https://github.com/jishengpeng/ControlSpeech .

ControlSpeech: Towards Simultaneous and Independent Zero-shot Speaker Cloning and Zero-shot Language Style Control

TL;DR

<3-5 sentence high-level summary> ControlSpeech introduces a first-of-its-kind TTS framework that simultaneously and independently achieves zero-shot timbre cloning and zero-shot style control by disentangling content, timbre, and style in a discrete codec space and guided by a Style Mixture Semantic Density module. The approach leverages a three-encoder architecture, a mask-based parallel codec generator, and a cross-attention fusion strategy, with SMSD modeling style descriptions as a Gaussian mixture to handle many-to-many mappings and to promote style diversity. Evaluations on the VccmDataset show competitive or state-of-the-art performance across controllability, timbre fidelity, and robustness, including out-of-domain and many-to-many scenarios, and a new dataset is released to support future research. The work also clarifies the limitations of prior approaches and provides open-source code and data to foster community development in controllable TTS.

Abstract

In this paper, we present ControlSpeech, a text-to-speech (TTS) system capable of fully cloning the speaker's voice and enabling arbitrary control and adjustment of speaking style. Prior zero-shot TTS models only mimic the speaker's voice without further control and adjustment capabilities while prior controllable TTS models cannot perform speaker-specific voice generation. Therefore, ControlSpeech focuses on a more challenging task: a TTS system with controllable timbre, content, and style at the same time. ControlSpeech takes speech prompts, content prompts, and style prompts as inputs and utilizes bidirectional attention and mask-based parallel decoding to capture codec representations corresponding to timbre, content, and style in a discrete decoupling codec space. Moreover, we analyze the many-to-many issue in textual style control and propose the Style Mixture Semantic Density (SMSD) module, which is based on Gaussian mixture density networks, to resolve this problem. To facilitate empirical validations, we make available a new style controllable dataset called VccmDataset. Our experimental results demonstrate that ControlSpeech exhibits comparable or state-of-the-art (SOTA) performance in terms of controllability, timbre similarity, audio quality, robustness, and generalizability. The relevant code and demo are available at https://github.com/jishengpeng/ControlSpeech .
Paper Structure (37 sections, 8 equations, 3 figures, 7 tables)

This paper contains 37 sections, 8 equations, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Text Prompt Based Controllable TTS
  4. Zero-shot TTS
  5. ControlSpeech
  6. Overall Architecture
  7. Codec Decoupling and Generation
  8. Decouple Content, Style, and Timbre
  9. Codec Generation Process
  10. The Style Mixture Semantic Density (SMSD) Module
  11. Training and Inference
  12. Experiments
  13. Experimental Setup
  14. VccmDataset.
  15. Baselines.
  16. ...and 22 more sections

Figures (3)

  • Figure 1: The voice prompt, the content description, and the style description correspond to the timbre, content, and style representations in the discrete codec space in the left panel . The right panel compares ControlSpeech with previous style-controllable TTS and zero-shot TTS systems. In this comparison, we use the amplitude and color of the waveform to represent the styleand timbre.
  • Figure 2: Figure (a) depicts the overall architecture of ControlSpeech, which is an encoder-decoder parallel disentangled codec generation model. Figure (b) provides a detailed illustration of the SMSD module, which addresses the many-to-many problem in style control by sampling from the style mixture semantic distribution and incorporating an additional noise perturbator. Figure (c) shows the process of the codec generator. Through masking, the codec can generate discrete codec representations in a fully non-autoregressive manner.
  • Figure 3: The t-SNE visualization of mixture density distribution after the SMSD module.