SimpleSpeech: Towards Simple and Efficient Text-to-Speech with Scalar Latent Transformer Diffusion Models

TL;DR

SimpleSpeech addresses the challenge of building an efficient TTS system using large-scale unlabeled speech without alignment information. It introduces SQ-Codec to map speech into a finite scalar latent space via scalar quantization and a transformer-based diffusion model operating in that space for non-autoregressive generation. Duration alignment is handled via sentence-level duration predicted by an LLM with in-context conditioning, enabling flexible outputs without phoneme-level timing. Across 4k hours of data, SimpleSpeech yields natural prosody and competitive or superior quality with faster generation and robust zero-shot voice cloning, outperforming several autoregressive and diffusion baselines.

Abstract

In this study, we propose a simple and efficient Non-Autoregressive (NAR) text-to-speech (TTS) system based on diffusion, named SimpleSpeech. Its simpleness shows in three aspects: (1) It can be trained on the speech-only dataset, without any alignment information; (2) It directly takes plain text as input and generates speech through an NAR way; (3) It tries to model speech in a finite and compact latent space, which alleviates the modeling difficulty of diffusion. More specifically, we propose a novel speech codec model (SQ-Codec) with scalar quantization, SQ-Codec effectively maps the complex speech signal into a finite and compact latent space, named scalar latent space. Benefits from SQ-Codec, we apply a novel transformer diffusion model in the scalar latent space of SQ-Codec. We train SimpleSpeech on 4k hours of a speech-only dataset, it shows natural prosody and voice cloning ability. Compared with previous large-scale TTS models, it presents significant speech quality and generation speed improvement. Demos are released.

Figures (1)

• Figure 1: (a) shows the overview of the SQ-Codec. (b) and (c) give the illustration of SimpleSpeech in training and inference.