DiffStyleTTS: Diffusion-based Hierarchical Prosody Modeling for Text-to-Speech with Diverse and Controllable Styles

TL;DR

DiffStyleTTS introduces a diffusion-based, multi-speaker TTS framework that hierarchically models prosody through coarse implicit style tokens and fine-grained explicit phoneme-level features. It uses a conditional diffusion module with two denoisers and classifier-free guidance, augmented by dynamic thresholding to stabilize guidance and prevent phoneme distortion. The model demonstrates higher naturalness and competitive speed compared with diffusion baselines, and shows strong prosodic transfer and controllability across three inference modes. This work advances flexible, controllable prosody in TTS, with practical implications for nuanced voice expression and speaker adaptation, while acknowledging remaining disentanglement challenges between prosody and timbre.

Abstract

Human speech exhibits rich and flexible prosodic variations. To address the one-to-many mapping problem from text to prosody in a reasonable and flexible manner, we propose DiffStyleTTS, a multi-speaker acoustic model based on a conditional diffusion module and an improved classifier-free guidance, which hierarchically models speech prosodic features, and controls different prosodic styles to guide prosody prediction. Experiments show that our method outperforms all baselines in naturalness and achieves superior synthesis speed compared to three diffusion-based baselines. Additionally, by adjusting the guiding scale, DiffStyleTTS effectively controls the guidance intensity of the synthetic prosody.

Figures (6)

• Figure 1: The model architecture of DiffStyleTTS. The LR refers to length regulator. The conditional diffusion module includes two denoisers $\varPsi_{\theta_1}(\bm{x}_t,t,\bm{y},\bm{c})$ and $\varPsi_{\theta_2}(\bm{x}_t,t,\bm{y})$.
• Figure 2: The diffusion process and reverse process of explicit prosodic features.
• Figure 3: An illustration of the training and inference processes of the conditional diffusion module based on classifier-free guidance.
• Figure 4: The Mel-spectrograms of a sentence synthesized with $\gamma=0$ (top), 0.4 (middle) and 0.7 (bottom) respectively.
• Figure 5: The t-SNE visualization results.