Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DiffCSS: Diverse and Expressive Conversational Speech Synthesis with Diffusion Models

Weihao wu, Zhiwei Lin, Yixuan Zhou, Jingbei Li, Rui Niu, Qinghua Wu, Songjun Cao, Long Ma, Zhiyong Wu

TL;DR

This work tackles the limitations of deterministic prosody and traditional TTS in conversational speech synthesis by introducing DiffCSS, a framework that blends diffusion-based prosody generation with a language-model-backed TTS backbone. It uses a diffusion-based context-aware prosody predictor to sample diverse, contextually appropriate embeddings from multimodal conversational context, which are then consumed by a prosody-enhanced ParlerTTS backbone to synthesize speech. The key contributions include the first application of diffusion models to CSS for prosody, a two-stage training strategy, and a cross-attention mechanism to derive fixed-length prosody embeddings. Experimental results on LibriTTS-R and DailyTalk show improvements in expressiveness, coherence, and prosody diversity over strong baselines, highlighting DiffCSS's potential for more natural and varied conversational speech.

Abstract

Conversational speech synthesis (CSS) aims to synthesize both contextually appropriate and expressive speech, and considerable efforts have been made to enhance the understanding of conversational context. However, existing CSS systems are limited to deterministic prediction, overlooking the diversity of potential responses. Moreover, they rarely employ language model (LM)-based TTS backbones, limiting the naturalness and quality of synthesized speech. To address these issues, in this paper, we propose DiffCSS, an innovative CSS framework that leverages diffusion models and an LM-based TTS backbone to generate diverse, expressive, and contextually coherent speech. A diffusion-based context-aware prosody predictor is proposed to sample diverse prosody embeddings conditioned on multimodal conversational context. Then a prosody-controllable LM-based TTS backbone is developed to synthesize high-quality speech with sampled prosody embeddings. Experimental results demonstrate that the synthesized speech from DiffCSS is more diverse, contextually coherent, and expressive than existing CSS systems

DiffCSS: Diverse and Expressive Conversational Speech Synthesis with Diffusion Models

TL;DR

This work tackles the limitations of deterministic prosody and traditional TTS in conversational speech synthesis by introducing DiffCSS, a framework that blends diffusion-based prosody generation with a language-model-backed TTS backbone. It uses a diffusion-based context-aware prosody predictor to sample diverse, contextually appropriate embeddings from multimodal conversational context, which are then consumed by a prosody-enhanced ParlerTTS backbone to synthesize speech. The key contributions include the first application of diffusion models to CSS for prosody, a two-stage training strategy, and a cross-attention mechanism to derive fixed-length prosody embeddings. Experimental results on LibriTTS-R and DailyTalk show improvements in expressiveness, coherence, and prosody diversity over strong baselines, highlighting DiffCSS's potential for more natural and varied conversational speech.

Abstract

Conversational speech synthesis (CSS) aims to synthesize both contextually appropriate and expressive speech, and considerable efforts have been made to enhance the understanding of conversational context. However, existing CSS systems are limited to deterministic prediction, overlooking the diversity of potential responses. Moreover, they rarely employ language model (LM)-based TTS backbones, limiting the naturalness and quality of synthesized speech. To address these issues, in this paper, we propose DiffCSS, an innovative CSS framework that leverages diffusion models and an LM-based TTS backbone to generate diverse, expressive, and contextually coherent speech. A diffusion-based context-aware prosody predictor is proposed to sample diverse prosody embeddings conditioned on multimodal conversational context. Then a prosody-controllable LM-based TTS backbone is developed to synthesize high-quality speech with sampled prosody embeddings. Experimental results demonstrate that the synthesized speech from DiffCSS is more diverse, contextually coherent, and expressive than existing CSS systems

Paper Structure

This paper contains 12 sections, 4 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Prosody-enhanced ParlerTTS
  4. Diffusion-based context-aware prosody predictor
  5. Training Strategy and Inference Procedure
  6. Experiments
  7. Training setup
  8. Baseline Models
  9. Subjective Evaluation
  10. Objective Evaluation
  11. Ablation Study on multimodal context
  12. Conclusion

Figures (3)

  • Figure 1: Overall architecture of the proposed CSS framework
  • Figure 2: The structure of the diffusion-based context-aware prosody predictor
  • Figure 3: Distribution of prosody embeddings in predictions and ground truth. Each unique color corresponds to a specific cluster.