Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Codec Language Model-based Zero-Shot Spontaneous Style TTS System for CoVoC Challenge 2024

Shuoyi Zhou, Yixuan Zhou, Weiqin Li, Jun Chen, Runchuan Ye, Weihao Wu, Zijian Lin, Shun Lei, Zhiyong Wu

TL;DR

This work presents a zero-shot spontaneous style TTS system for CoVoC 2024, built on a LLaMA-based codec language model with a delay-pattern autoregressive framework and augmented by Classifier-Free Guidance to boost intelligibility. The model leverages MT5-based text embeddings, HuBERT+K-means semantic tokens, and DAC-based acoustic tokens, trained in staged pretraining on large corpora and fine-tuned on high-quality spontaneous data (HQ-Conversations) plus premium WenetSpeech4TTS. Data preprocessing removes noise and overlaps, while CFG during inference blends conditional and unconditional signals to improve synthesis quality. In CoVoC constrained track evaluations, the approach achieves state-of-the-art naturalness MOS ($3.80$), strong quality and speaker similarity, and solid robustness, demonstrating effective zero-shot spontaneous-style voice cloning in practical conversational scenarios.

Abstract

This paper describes the zero-shot spontaneous style TTS system for the ISCSLP 2024 Conversational Voice Clone Challenge (CoVoC). We propose a LLaMA-based codec language model with a delay pattern to achieve spontaneous style voice cloning. To improve speech intelligibility, we introduce the Classifier-Free Guidance (CFG) strategy in the language model to strengthen conditional guidance on token prediction. To generate high-quality utterances, we adopt effective data preprocessing operations and fine-tune our model with selected high-quality spontaneous speech data. The official evaluations in the CoVoC constrained track show that our system achieves the best speech naturalness MOS of 3.80 and obtains considerable speech quality and speaker similarity results.

The Codec Language Model-based Zero-Shot Spontaneous Style TTS System for CoVoC Challenge 2024

TL;DR

This work presents a zero-shot spontaneous style TTS system for CoVoC 2024, built on a LLaMA-based codec language model with a delay-pattern autoregressive framework and augmented by Classifier-Free Guidance to boost intelligibility. The model leverages MT5-based text embeddings, HuBERT+K-means semantic tokens, and DAC-based acoustic tokens, trained in staged pretraining on large corpora and fine-tuned on high-quality spontaneous data (HQ-Conversations) plus premium WenetSpeech4TTS. Data preprocessing removes noise and overlaps, while CFG during inference blends conditional and unconditional signals to improve synthesis quality. In CoVoC constrained track evaluations, the approach achieves state-of-the-art naturalness MOS (), strong quality and speaker similarity, and solid robustness, demonstrating effective zero-shot spontaneous-style voice cloning in practical conversational scenarios.

Abstract

This paper describes the zero-shot spontaneous style TTS system for the ISCSLP 2024 Conversational Voice Clone Challenge (CoVoC). We propose a LLaMA-based codec language model with a delay pattern to achieve spontaneous style voice cloning. To improve speech intelligibility, we introduce the Classifier-Free Guidance (CFG) strategy in the language model to strengthen conditional guidance on token prediction. To generate high-quality utterances, we adopt effective data preprocessing operations and fine-tune our model with selected high-quality spontaneous speech data. The official evaluations in the CoVoC constrained track show that our system achieves the best speech naturalness MOS of 3.80 and obtains considerable speech quality and speaker similarity results.

Paper Structure

This paper contains 15 sections, 6 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Data Preparation
  3. System
  4. Feature Representation
  5. Model Architecture
  6. Classifier-Free Guidance for Codec Language Model
  7. Experiments
  8. Model Configurations
  9. Training and Inference Details
  10. Results
  11. Evaluation metrics
  12. Evaluation results and discussion
  13. Case Study
  14. Conclusions
  15. Acknowledgements

Figures (2)

  • Figure 1: The model architecture in training procedure.
  • Figure 2: The mel-spectrograms of speeches synthesized by proposed model. The texts of these two samples are "嗯没有诶，如果你爬到过的话可以和我介绍一下" (English translation: "Um, no, I haven't. If you've climbed there before, you can tell me about it.") and "然后类似于啊这样的，嗯，不太满意的体验，啊还有很多。" (English translation: "And then, there were experiences like, uh, this, um, not very satisfying, uh, and there were many more.") respectively.