Parallel Synthesis for Autoregressive Speech Generation

TL;DR

This work addresses the inefficiency of autoregressive neural vocoders by relocating autoregression from the time domain to frequency and bit-precision domains. It introduces frequency-wise autoregressive generation (FAR) and bit-wise autoregressive generation (BAR), combined with a post-filter for high-fidelity posterior sampling, enabling fixed, small iteration counts and real-time synthesis on GPUs. Empirical results show competitive objective metrics and strong human-perceived quality, along with robust generalization to unseen speakers and high-fidelity datasets. The approach narrows the performance gap between autoregressive and non-autoregressive vocoders while preserving AR advantages such as temporal coherence and stability, with practical implications for real-time TTS, VC, and related speech tasks.

Abstract

Autoregressive neural vocoders have achieved outstanding performance in speech synthesis tasks such as text-to-speech and voice conversion. An autoregressive vocoder predicts a sample at some time step conditioned on those at previous time steps. Though it synthesizes natural human speech, the iterative generation inevitably makes the synthesis time proportional to the utterance length, leading to low efficiency. Many works were dedicated to generating the whole speech sequence in parallel and proposed GAN-based, flow-based, and score-based vocoders. This paper proposed a new thought for the autoregressive generation. Instead of iteratively predicting samples in a time sequence, the proposed model performs frequency-wise autoregressive generation (FAR) and bit-wise autoregressive generation (BAR) to synthesize speech. In FAR, a speech utterance is split into frequency subbands, and a subband is generated conditioned on the previously generated one. Similarly, in BAR, an 8-bit quantized signal is generated iteratively from the first bit. By redesigning the autoregressive method to compute in domains other than the time domain, the number of iterations in the proposed model is no longer proportional to the utterance length but to the number of subbands/bits, significantly increasing inference efficiency. Besides, a post-filter is employed to sample signals from output posteriors; its training objective is designed based on the characteristics of the proposed methods. Experimental results show that the proposed model can synthesize speech faster than real-time without GPU acceleration. Compared with baseline vocoders, the proposed model achieves better MUSHRA results and shows good generalization ability for unseen speakers and 44 kHz speech.

Figures (15)

• Figure 1: Overview of different autoregressive methods and their orders of generation. (a) Conventional autoregressive generation. Samples at different time steps are generated sequentially. (b) Frequency-wise autoregressive generation (FAR). Subbands are first generated autoregressively and combined to form the full-band waveform. (c) Bit-wise autoregressive generation (BAR). The spectrograms in (b) and Mel-spectrograms in (c) are only for visualization and not for generation.
• Figure 2: (a) Conventional autoregressive model. Each blue circle represents a scalar. (b) Proposed autoregressive model. The speech is generated iteratively in the frequency domain or the bit precision domain. Each green block is the model illustrated in Fig. \ref{['fig:arc']} (a), and each orange block represents a time series.
• Figure 3: Frequency-wise autoregressive generation.
• Figure 4: Bit-wise autoregressive generation.
• Figure 5: (a) Overview and detailed block diagram of the proposed model. The model predicts the $i$th subband $x^{i}$ conditioned on the previous subband $x^{i-1}$. $h^{i}$ is the hidden state, and $f^{i}$ is the upsampled acoustic feature. BAR is integrated in each FAR prediction, and $b^{i,1}, b^{i,2}, b^{i,3}$ are the first three bits of $x^{i}$. Channel sizes are shown in gray and parenthesized. (b) WN module.