TalkNet 2: Non-Autoregressive Depth-Wise Separable Convolutional Model for Speech Synthesis with Explicit Pitch and Duration Prediction

TL;DR

TalkNet tackles shortcomings of autoregressive TTS by introducing a non-autoregressive, fully convolutional pipeline with explicit duration and pitch modeling. It decomposes TTS into three CNN modules—grapheme duration predictor, pitch predictor, and mel-spectrogram generator—each based on depthwise separable convolutions, enabling parallel training and fast inference without teacher models. On LJSpeech, TalkNet achieves MOS nearly on par with autoregressive systems while using only $13.2$M parameters, and demonstrates substantial speedups during inference (e.g., approximate real-time factors of $0.0076$ for batch 1 and $0.000395$ for batch 32). This efficiency and robustness make TalkNet attractive for embedded speech synthesis and streaming, with plans to release the implementation in the NeMo toolkit.

Abstract

We propose TalkNet, a non-autoregressive convolutional neural model for speech synthesis with explicit pitch and duration prediction. The model consists of three feed-forward convolutional networks. The first network predicts grapheme durations. An input text is expanded by repeating each symbol according to the predicted duration. The second network predicts pitch value for every mel frame. The third network generates a mel-spectrogram from the expanded text conditioned on predicted pitch. All networks are based on 1D depth-wise separable convolutional architecture. The explicit duration prediction eliminates word skipping and repeating. The quality of the generated speech nearly matches the best auto-regressive models - TalkNet trained on the LJSpeech dataset got MOS 4.08. The model has only 13.2M parameters, almost 2x less than the present state-of-the-art text-to-speech models. The non-autoregressive architecture allows for fast training and inference. The small model size and fast inference make the TalkNet an attractive candidate for embedded speech synthesis.

Figures (5)

• Figure 1: TalkNet converts text to speech, using a grapheme duration predictor, pitch predictor and a mel-spectrogram generator. We use $\sim$ to denote the blank symbol.
• Figure 2: Grapheme duration prediction.
• Figure 3: Basic QuartzNet block. Both the grapheme duration predictor and the mel-spectrogram generator are 1D time-channel convolutional networks based on QuartzNet quartznet.
• Figure 4: The duration distribution for graphemes (left) and for blanks (right) based on CTC output for the LJSpeech dataset.
• Figure 5: Pitch predictor loss is a sum of two heads: one head is binary cross-entropy (BCE) for non-voiced prediction, another head is mean squared error (MSE) for non-zero values.