DiscreTalk: Text-to-Speech as a Machine Translation Problem
Tomoki Hayashi, Shinji Watanabe
TL;DR
This work reframes text-to-speech as a translation problem by mapping text to a sequence of discrete speech symbols learned via a non-autoregressive GAN-based VQ-VAE, followed by autoregressive Transformer-NMT decoding to those symbols. The discrete-symbol representation enables leveraging NMT/ASR techniques such as beam search and subword units, while avoiding hand-picked acoustic features and reducing over-smoothing. Empirical results on JSUT show the approach surpasses a Transformer-TTS baseline in naturalness and approaches the reconstruction upper bound, with subword units and a suitable downsampling factor providing the strongest gains. The study also identifies a trade-off between symbol resolution and articulation and suggests directions for scaling to multi-speaker and larger datasets, as well as exploring fully non-autoregressive variants.
Abstract
This paper proposes a new end-to-end text-to-speech (E2E-TTS) model based on neural machine translation (NMT). The proposed model consists of two components; a non-autoregressive vector quantized variational autoencoder (VQ-VAE) model and an autoregressive Transformer-NMT model. The VQ-VAE model learns a mapping function from a speech waveform into a sequence of discrete symbols, and then the Transformer-NMT model is trained to estimate this discrete symbol sequence from a given input text. Since the VQ-VAE model can learn such a mapping in a fully-data-driven manner, we do not need to consider hyperparameters of the feature extraction required in the conventional E2E-TTS models. Thanks to the use of discrete symbols, we can use various techniques developed in NMT and automatic speech recognition (ASR) such as beam search, subword units, and fusions with a language model. Furthermore, we can avoid an over smoothing problem of predicted features, which is one of the common issues in TTS. The experimental evaluation with the JSUT corpus shows that the proposed method outperforms the conventional Transformer-TTS model with a non-autoregressive neural vocoder in naturalness, achieving the performance comparable to the reconstruction of the VQ-VAE model.