DQR-TTS: Semi-supervised Text-to-speech Synthesis with Dynamic Quantized Representation

TL;DR

This work addresses the data scarcity challenge in neural text-to-speech by proposing DQR-TTS, a semi-supervised TTS framework that learns from both paired and unpaired data. It integrates a dynamic quantized representation within a sequential autoencoder, where a trainable codebook is supervised by paired data and expanded during training via unpaired data using a distance-based update strategy and sharped pseudo-labels. The method employs frame-synchronized quantization and phoneme-synchronized segmentation to produce phoneme-level representations, and trains with a joint objective that combines reconstruction, recognition via CTC, and decoder losses: $\mathcal{L}_{total}=\mathcal{L}_{recon}+\alpha_1\mathcal{L}_{recog}+\alpha_2\mathcal{L}_{dec}$. Experiments on LJSpeech show that with less than 120 minutes of paired data, DQR-TTS outperforms several baselines in both MOS and MCD while effectively leveraging unpaired data, indicating strong potential for low-resource TTS deployment.

Abstract

Most existing neural-based text-to-speech methods rely on extensive datasets and face challenges under low-resource condition. In this paper, we introduce a novel semi-supervised text-to-speech synthesis model that learns from both paired and unpaired data to address this challenge. The key component of the proposed model is a dynamic quantized representation module, which is integrated into a sequential autoencoder. When given paired data, the module incorporates a trainable codebook that learns quantized representations under the supervision of the paired data. However, due to the limited paired data in low-resource scenario, these paired data are difficult to cover all phonemes. Then unpaired data is fed to expand the dynamic codebook by adding quantized representation vectors that are sufficiently distant from the existing ones during training. Experiments show that with less than 120 minutes of paired data, the proposed method outperforms existing methods in both subjective and objective metrics.

Figures (2)

• Figure 1: Pipeline of DQR-TTS. The training process consists of three steps: (1) The paired data is fed into the network to compare the distance between each continuous vector and codeword, and then replace each vector with the nearest codeword respectively. Then Mapping the codewords to phonemes according to the labels. (2) Generate pseudo labels via pre-trained ASR and execute the dynamic codebook updating strategy with unpaired data. (3) Mapping the unpaired codewords to phonemes according to the pseudo labels. Then the paired data and unpaired data with pseudo labels are jointly utilized to train the model.
• Figure 2: Four groups of training data with different P/U ratios. The purple bars represent training with P data only, while the orange bars represent the results from training with P&U data.