Single-stage TTS with Masked Audio Token Modeling and Semantic Knowledge Distillation
Gerard I. Gállego, Roy Fejgin, Chunghsin Yeh, Xiaoyu Liu, Gautam Bhattacharya
TL;DR
The paper addresses the challenge of achieving high-quality, intelligible TTS without a separate semantic-to-audio stage by proposing NARSiS, a non-autoregressive single-stage TTS model that jointly models semantic and acoustic tokens. Semantic Knowledge Distillation (SKD) injects semantic information during training via HuBERT-derived representations, enabling a unified model to leverage high-level semantics without adding inference stages. Empirical results show substantial gains in intelligibility and speaker similarity over a single-stage baseline, with continuous semantic features outperforming discrete ones and the best SKD variant (NARSiSavg) approaching two-stage systems in several metrics while delivering up to 50% faster inference. Although two-stage systems still hold an edge in intelligibility, the proposed approach narrows the gap and demonstrates the viability of efficient, high-quality single-stage TTS using SKD and MATM.
Abstract
Audio token modeling has become a powerful framework for speech synthesis, with two-stage approaches employing semantic tokens remaining prevalent. In this paper, we aim to simplify this process by introducing a semantic knowledge distillation method that enables high-quality speech generation in a single stage. Our proposed model improves speech quality, intelligibility, and speaker similarity compared to a single-stage baseline. Although two-stage systems still lead in intelligibility, our model significantly narrows the gap while delivering comparable speech quality. These findings showcase the potential of single-stage models to achieve efficient, high-quality TTS with a more compact and streamlined architecture.