DualCodec: A Low-Frame-Rate, Semantically-Enhanced Neural Audio Codec for Speech Generation

TL;DR

DualCodec tackles the trade-off between frame rate and audio quality in neural audio codecs for LM-based speech generation by introducing a dual-stream encoding that fuses SSL-derived semantic information with waveform-based DAC representations. It achieves low-frame-rate operation ($12.5$ Hz and $25$ Hz) while preserving high audio quality through larger RVQ codebooks and dual encoding, outperforming state-of-the-art codecs such as Mimi Codec, SpeechTokenizer, DAC, and Encodec on semantic content and audio reconstruction metrics. The method yields substantial gains in TTS quality and efficiency, with the $25$ Hz configuration delivering the best semantic accuracy and perceptual quality, and the $12.5$ Hz version offering faster inference at a manageable quality trade-off. Extensive experiments across semantic analysis, audio quality, and Seed-TTS-Eval demonstrate the approach's effectiveness, and the work provides open-source DualCodec TTS systems for practical deployment.

Abstract

Neural audio codecs form the foundational building blocks for language model (LM)-based speech generation. Typically, there is a trade-off between frame rate and audio quality. This study introduces a low-frame-rate, semantically enhanced codec model. Existing approaches distill semantically rich self-supervised (SSL) representations into the first-layer codec tokens. This work proposes DualCodec, a dual-stream encoding approach that integrates SSL and waveform representations within an end-to-end codec framework. In this setting, DualCodec enhances the semantic information in the first-layer codec and enables the codec system to maintain high audio quality while operating at a low frame rate. Note that a low-frame-rate codec improves the efficiency of speech generation. Experimental results on audio codec and speech generation tasks confirm the effectiveness of the proposed DualCodec compared to state-of-the-art codec systems, such as Mimi Codec, SpeechTokenizer, DAC, and Encodec. Demos are available at: https://dualcodec.github.io, code is available at: https://github.com/jiaqili3/DualCodec

Figures (5)

• Figure 1: The dual encoding method for neural audio codecs. The upper stream is SSL encoding, and the lower stream is waveform encoding. Given a speech input, the SSL feature is obtained from a pretrained w2v-bert-2 seamless model and then encoded as the codec's first-layer token (RVQ-1). The remaining RVQ layers encodes the residual between the waveform feature and the RVQ-1 feature, and outputs audio. The framework is trained end-to-end requiring an additional L2 SSL feature loss in addition to codec training losses.
• Figure 2: Architecture of VALL-E TTS.
• Figure 3: Architecture of AR+SoundStorm TTS.
• Figure 4: The architecture of a simple FlattenedAR TTS. It models a flattened sequence of four layer RVQ tokens from DualCodec 12.5Hz.
• Figure 5: Architecture of MaskGCT TTS. Compared to the AR+SoundStorm TTS, its first stage instead uses the masked token prediction paradigm.