SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General Sound

TL;DR

SemantiCodec tackles the challenge of semantically meaningful, ultra-low-bitrate audio coding by decoupling semantic content from acoustic detail via a dual-encoder architecture and a diffusion-based decoder. By combining AudioMAE-based semantic clustering with an acoustic BiLSTM encoder and a latent diffusion model, it achieves 25–100 tokens per second (0.31–1.40 kbps) while outperforming state-of-the-art baselines in reconstruction and delivering richer semantic information for downstream audio-language tasks. Extensive experiments across speech, music, and general sounds, plus HEAR-based semantic evaluations and MUSHRA tests, demonstrate robust reconstruction quality and strong semantic content even at very low bitrates. The work highlights the potential of diffusion-driven decoders for semantic audio codecs and paves the way for more efficient, language-model-friendly audio representations.

Abstract

Large language models (LLMs) have significantly advanced audio processing through audio codecs that convert audio into discrete tokens, enabling the application of language modelling techniques to audio data. However, traditional codecs often operate at high bitrates or within narrow domains such as speech and lack the semantic clues required for efficient language modelling. Addressing these challenges, we introduce SemantiCodec, a novel codec designed to compress audio into fewer than a hundred tokens per second across diverse audio types, including speech, general sound, and music, without compromising quality. SemantiCodec features a dual-encoder architecture: a semantic encoder using a self-supervised pre-trained Audio Masked Autoencoder (AudioMAE), discretized using k-means clustering on extensive audio data, and an acoustic encoder to capture the remaining details. The semantic and acoustic encoder outputs are used to reconstruct audio via a diffusion-model-based decoder. SemantiCodec is presented in three variants with token rates of 25, 50, and 100 per second, supporting a range of ultra-low bit rates between 0.31 kbps and 1.40 kbps. Experimental results demonstrate that SemantiCodec significantly outperforms the state-of-the-art Descript codec on reconstruction quality. Our results also suggest that SemantiCodec contains significantly richer semantic information than all evaluated state-of-the-art audio codecs, even at significantly lower bitrates. Our code and demos are available at https://haoheliu.github.io/SemantiCodec/.

Figures (7)

• Figure 1: Comparison between SemantiCodec and state-of-the-art codecs. Higher values on the horizontal and vertical axis indicate better reconstruction quality and semantic information, respectively. The size of circles indicates bitrates, where smaller ones denote lower bitrates.
• Figure 2: SemantiCodec architecture. For an input audio clip, quantized semantic representation ${\bm{E}}_{s}$ is obtained via a pre-computed codebook using k-means clustering on the AudioMAE embeddings. Then ${\bm{Y}}$ and ${\bm{E}}_\text{s}$ are concatenated and fed to a residual encoder to complement acoustic details, which is discretized to ${\bm{E}}_{a}$ by a vector quantization module. SemantiCodec encoder output ${\bm{E}}$ is obtained by concatenating ${\bm{E}}_{s}$ and ${\bm{E}}_{a}$. A latent diffusion model is trained to generate the original audio clip conditioned on ${\bm{E}}$. The snowflake and fire symbols denote frozen and learnable parameters, respectively.
• Figure 3: The average MUSHRA test score on our curated evaluation set. Our proposed SemantiCodec outperforms the baseline models even with a significantly lower bitrate. For fair comparisons, DAC and HiFi-Codec are not included as they use much higher bit rates.
• Figure 4: The MUSHRA test score on different domains.
• Figure 5: The log-STFT spectrogram of the ground truth audio and the reconstruction audio with different audio codecs. DAC, EC, HC, and SC are the descript codec, Encodec, HiFi-Codec, and SemantiCodec, respectively.