GSA-TTS : Toward Zero-Shot Speech Synthesis based on Gradual Style Adaptor
Seokgi Lee, Jungjun Kim
TL;DR
The paper tackles zero-shot speech synthesis by learning a generalizable, hierarchical style representation. It introduces GSA-TTS, which combines a Local Style Encoder and a Global Style Encoder guided by ASR-based style segmentation to extract word-level, noise-free style segments from reference audio, and fuses them with self-attention to produce a robust global style condition. The approach includes a CLN-based conditioning mechanism and extensive ablations, showing improved naturalness, speaker similarity, and intelligibility over FastPitch baselines, along with interpretable controllability via POS-aware attention. The results suggest that gradual, multi-level style encoding reduces content leakage and enhances the quality of unseen-speaker synthesis, offering a practical, plug-and-play improvement for zero-shot TTS systems with potential for targeted style control.
Abstract
We present the gradual style adaptor TTS (GSA-TTS) with a novel style encoder that gradually encodes speaking styles from an acoustic reference for zero-shot speech synthesis. GSA first captures the local style of each semantic sound unit. Then the local styles are combined by self-attention to obtain a global style condition. This semantic and hierarchical encoding strategy provides a robust and rich style representation for an acoustic model. We test GSA-TTS on unseen speakers and obtain promising results regarding naturalness, speaker similarity, and intelligibility. Additionally, we explore the potential of GSA in terms of interpretability and controllability, which stems from its hierarchical structure.