U-Codec: Ultra Low Frame-rate Neural Speech Codec for Fast High-fidelity Speech Generation

TL;DR

U-Codec tackles the challenge of extreme speech compression by operating at an ultra-low frame rate of 5 Hz, enabling fast, high-fidelity speech generation suitable for LLM-based TTS. It combines a Transformer-based inter-frame dependency module with a multi-layer, factorized RVQ (FRVQ) and a CodecFormer global-local architecture to model long-range dependencies while maintaining spectral detail. Through extensive codec and TTS experiments on large multilingual corpora, the approach achieves roughly a 3× inference speedup over higher-frame-rate codecs while maintaining competitive intelligibility and naturalness, even when using up to 100 RVQ layers. These findings demonstrate the feasibility of 5 Hz discrete speech tokens for rapid, high-quality speech synthesis and offer a scalable path for integrating discrete speech representations with LLM-based generation systems.

Abstract

We propose \textbf{U-Codec}, an \textbf{U}ltra low frame-rate neural speech \textbf{Codec} that achieves high-fidelity reconstruction and fast speech generation at an extremely low frame-rate of 5Hz (5 frames per second). Extreme compression at 5Hz typically leads to severe intelligibility and spectral detail loss, we introduce a Transformer-based inter-frame long-term dependency module and systematically explore residual vector quantization (RVQ) depth and codebook size to identify optimal configurations. Moreover, we apply U-Codec into a large language model (LLM)-based auto-regressive TTS model, which leverages global and local hierarchical architecture to effectively capture dependencies across multi-layer tokens. We extend LLM-based TTS from 3-layer RVQ at 50Hz to 32-layer RVQ at 5Hz. Experimental results demonstrate that U-Codec improves LLM-based TTS inference speed by around 3 $\times$ over high-frame-rate codecs while maintaining similarity and naturalness. These results validate the feasibility of using highly compressed 5Hz discrete tokens for fast and high-fidelity speech synthesis.

Figures (3)

• Figure 1: Reconstruct quality on varying the frame rate of different codecs, where bubble size indicates the number of RVQ layers. Compared to previous systems, such as EnCodec, Mimi, and DAC operating at higher frame rates, our proposed codec achieves competitive PESQ performance under the ultra low frame rate (5Hz).
• Figure 2: Architecture and training of our neural speech codec at 5Hz. The encoder (left) consists of convolutional layers followed by a Transformer to capture long-term dependencies. Latent features are quantized through residual vector quantization (RVQ) and optimized for high-fidelity reconstruction. The decoder (right) mirrors the encoder to synthesize the output waveform. We use reconstruction loss and adversarial loss during training.
• Figure 3: Hierarchical global-local Transformer architecture. Each input sequence is divided into patches, where $z^{k}_{N}$ denotes the $k-th$ token in the $t-th$ patch.