VQVAE Unsupervised Unit Discovery and Multi-scale Code2Spec Inverter for Zerospeech Challenge 2019

TL;DR

The paper tackles zero-resource speech synthesis by combining a frame-based VQ-VAE for unsupervised unit discovery with a multi-scale Code2Spec inverter to generate target-speaker magnitude spectrograms. It systematically compares VQ-VAE against direct MFCC/Mel features, K-Means, and GMM baselines, and demonstrates that the VQ-VAE with Code2Spec yields a favorable ABX/bit-rate trade-off while improving intelligibility (CER, MOS). The approach is evaluated on English and a surprise Austronesian language, showing substantial gains over the ZeroSpeech 2019 baseline and topline, and highlighting the potential of end-to-end unsupervised unit discovery for zero-resource TTS. The work also discusses limitations and future directions, including exploring WaveNet/GAN-based refinements and better speech quality enhancements.

Abstract

We describe our submitted system for the ZeroSpeech Challenge 2019. The current challenge theme addresses the difficulty of constructing a speech synthesizer without any text or phonetic labels and requires a system that can (1) discover subword units in an unsupervised way, and (2) synthesize the speech with a target speaker's voice. Moreover, the system should also balance the discrimination score ABX, the bit-rate compression rate, and the naturalness and the intelligibility of the constructed voice. To tackle these problems and achieve the best trade-off, we utilize a vector quantized variational autoencoder (VQ-VAE) and a multi-scale codebook-to-spectrogram (Code2Spec) inverter trained by mean square error and adversarial loss. The VQ-VAE extracts the speech to a latent space, forces itself to map it into the nearest codebook and produces compressed representation. Next, the inverter generates a magnitude spectrogram to the target voice, given the codebook vectors from VQ-VAE. In our experiments, we also investigated several other clustering algorithms, including K-Means and GMM, and compared them with the VQ-VAE result on ABX scores and bit rates. Our proposed approach significantly improved the intelligibility (in CER), the MOS, and discrimination ABX scores compared to the official ZeroSpeech 2019 baseline or even the topline.

Figures (4)

• Figure 1: Speech chain model consists of an ASR and a TTS module. Given speech features $\mathbf{x}$, the ASR generates a transcription. The TTS reconstructs the speech features based on the transcription and calculates the reconstruction loss ${\cal L}^{rec}_{tts}$.
• Figure 2: Conditional VQ-VAEs consist of four main modules: encoder $q_\theta(z|x)$, decoder $p_\phi(x|z,s)$, codebooks $E =[e_1,..,e_K]$, and speaker embedding $V = [v_1,..,v_L]$.
• Figure 3: Code-to-speech inverter: given a sequence of codebook $\left[e[1], e[2],.., e[T_z]\right]$, we duplicate each codebook based on compression ratio $r=4$ and apply multiple layers of multi-scale 1D convolution + LeakyReLU activation function to predict the target voice linear spectrogram $\hat{M}$.
• Figure 4: Left: VQ-VAE encoder and decoder architecture with 4x time reduction (based on stride size in encoder layer). Right: Code2Spec architecture. Definition: K is kernel size, C is output channel, S is stride size, and T is input frame length. K:3x3 denotes 2D convolution with 3x3 kernel size across time and frequency axis, K:[1,3,5,7] denotes 1D convolution with 4 different kernel size (1, 3, 5, 7) across time-axis.