Improving X-Codec-2.0 for Multi-Lingual Speech: 25 Hz Latent Rate and 24 kHz Sampling
Husein Zolkepli
TL;DR
The paper tackles the limitation of X-Codec-2.0’s 50 Hz latent rate and 16 kHz sampling by introducing a simple temporal pooling method and a larger encoder hop to achieve 25 Hz latent rate while raising the output to 24 kHz. By freezing the semantic encoder and codec encoder and only fine-tuning the decoder, the approach preserves the original architecture and training dynamics, while decoder-weight interpolation adapts to the new token dimensionality. On the multilingual Common Voice 17 dataset, the proposed 25 Hz/24 kHz configuration yields a +0.29 MOS improvement (via UTMOSv2) and sets a new state-of-the-art for codecs operating at 25 Hz. The results demonstrate that targeted, minimal architectural refinements can deliver perceptually meaningful gains in neural audio codecs while maintaining compatibility with LLM-based speech modeling in multilingual settings.
Abstract
X-Codec-2.0 has shown strong performance in neural audio compression and multilingual speech modeling, operating at a 50 Hz latent rate and a 16 kHz sampling rate using frozen HuBERT features. While effective, this configuration limits temporal efficiency and audio fidelity. In this work, we explore a simple and effective modification by introducing additional pooling and increasing the decoder hop size. This reduces the latent rate from 50 Hz to 25 Hz and simultaneously raises the output sampling rate from 16 kHz to 24 kHz, improving efficiency and perceptual quality without altering the core architecture. Evaluated on the multilingual Common Voice 17 test set, the proposed configuration achieves a 0.29 MOS improvement over the original X-Codec-2.0 baseline based on UTMOSv2, and attains the best reported performance among all codecs operating at 25 Hz. The source code, checkpoints, and generation comparisons are released at \href{https://huggingface.co/Scicom-intl/xcodec2-25TPS-24k}{https://huggingface.co/Scicom-intl/xcodec2-25TPS-24k}.