HALL-E: Hierarchical Neural Codec Language Model for Minute-Long Zero-Shot Text-to-Speech Synthesis

TL;DR

Two novel post-training approaches to reduce the frame rate of pre-trained NAC models are introduced, enabling the stable minitue-long speech synthesis in a single inference step and to promote TTS research.

Abstract

Recently, Text-to-speech (TTS) models based on large language models (LLMs) that translate natural language text into sequences of discrete audio tokens have gained great research attention, with advances in neural audio codec (NAC) models using residual vector quantization (RVQ). However, long-form speech synthesis remains a significant challenge due to the high frame rate, which increases the length of audio tokens and makes it difficult for autoregressive language models to generate audio tokens for even a minute of speech. To address this challenge, this paper introduces two novel post-training approaches: 1) Multi-Resolution Requantization (MReQ) and 2) HALL-E. MReQ is a framework to reduce the frame rate of pre-trained NAC models. Specifically, it incorporates multi-resolution residual vector quantization (MRVQ) module that hierarchically reorganizes discrete audio tokens through teacher-student distillation. HALL-E is an LLM-based TTS model designed to predict hierarchical tokens of MReQ. Specifically, it incorporates the technique of using MRVQ sub-modules and continues training from a pre-trained LLM-based TTS model. Furthermore, to promote TTS research, we create MinutesSpeech, a new benchmark dataset consisting of 40k hours of filtered speech data for training and evaluating speech synthesis ranging from 3s up to 180s. In experiments, we demonstrated the effectiveness of our approaches by applying our post-training framework to VALL-E. We achieved the frame rate down to as low as 8 Hz, enabling the stable minitue-long speech synthesis in a single inference step. Audio samples, dataset, codes and pre-trained models are available at https://yutonishimura-v2.github.io/HALL-E_DEMO/.

Figures (7)

• Figure 1: Preliminary experiments on varying the frame rate of Encodec.
• Figure 2: MReQ post-training based on teacher-student distillation. (a) Pre-trained RVQ module is used to extract teacher embeddings ${\bm{h}_{t}}$. (b) MRVQ module consists of multiple LRVQ blocks and learns to reduce the frame rates. Student embeddings ${\bm{h}_{s}}$ are extracted. (c) Each LRVQ block consists of a pre-quantizer $\mathrm{PreQ}$, a sub-encoder $E$, a main quantizer $\mathrm{Quant}$, a sub-decoder $D$, and a post-quantizer $\mathrm{PostQ}$ to reduce frame rate from $s_{0}$ to $s_{k}$.
• Figure 3: HALL-E architecture. (a) AR model generates a low frame-rate token sequence $\hat{\bm{b}}_{1}$. (b) NAR model predicts $\hat{\bm{b}}_{k+1}$ from $\hat{\bm{b}}_{k}$ iteratively by utilizing frozen sub-modules of MRVQ.
• Figure 4: Duration and word count distributions.
• Figure 5: Samples of generated waveforms