Voiced-Aware Style Extraction and Style Direction Adjustment for Expressive Text-to-Speech

TL;DR

Expressive TTS remains challenging due to the need to transfer nuanced speaking styles while avoiding content leakage. The authors propose Spotlight-TTS, which combines voiced-aware style extraction with a style direction adjustment mechanism, and uses a FastSpeech 2-based architecture with a two-part style encoder and biased self-attention in an unvoiced filler module. The approach introduces a rotation trick for residual vector quantization and two losses—style disentanglement and style preserving—to disentangle content while preserving prosody. Experimental results on the Emotional Speech Dataset show Spotlight-TTS achieving state-of-the-art naturalness and style similarity, strong pronunciation accuracy, and robust style transfer in both parallel and non-parallel conditions, indicating substantial potential for practical expressive TTS deployments.

Abstract

Recent advances in expressive text-to-speech (TTS) have introduced diverse methods based on style embedding extracted from reference speech. However, synthesizing high-quality expressive speech remains challenging. We propose SpotlightTTS, which exclusively emphasizes style via voiced-aware style extraction and style direction adjustment. Voiced-aware style extraction focuses on voiced regions highly related to style while maintaining continuity across different speech regions to improve expressiveness. We adjust the direction of the extracted style for optimal integration into the TTS model, which improves speech quality. Experimental results demonstrate that Spotlight-TTS achieves superior performance compared to baseline models in terms of expressiveness, overall speech quality, and style transfer capability.

Figures (3)

• Figure 1: Overall architecture of Spotlight-TTS. $E_c$, $E_g$, and $E_s$ denote the content embedding, global style embedding, and style embedding respectively.
• Figure 2: Detailed architecture of the style encoder. The encoder processes voiced regions through RT-RVQ (Rotation Trick with Residual Vector Quantization) and fills unvoiced regions using the Unvoiced Filler module with biased self-attention. Yellow regions indicate voiced frames, gray regions indicate unvoiced frames, and dotted boxes represent initialized mask code embeddings. $e$ and $q$ represent the input feature and quantized vector respectively.
• Figure 3: Conceptual illustration of style direction adjustment. The figure shows how the direction of the style vector (orange) is adjusted in the embedding space. The style disentanglement loss $\mathcal{L}_{sd}$ encourages orthogonality between the style vector and content vector (blue), while the style preserving loss $\mathcal{L}_{sp}$ maintains alignment with the prosody vector (green).