CodeSep: Low-Bitrate Codec-Driven Speech Separation with Base-Token Disentanglement and Auxiliary-Token Serial Prediction

TL;DR

This work tackles joint speech separation and low-bitrate compression by producing discrete tokens for efficient transmission or storage. CodeSep integrates a residual vector quantizer–based plain codec (MDCTCodec), a base-token disentanglement module, and parallel auxiliary-token serial prediction modules to separate and compress mixed speech. Base tokens carry speaker information and are transmitted, while auxiliary tokens refine separation during decoding, enabling reconstruction at a very low bitrate of 1 kbps. Experiments on LibriMix demonstrate strong separation quality and speaker similarity at low bitrate, with ablations validating the design choices (ACBG and teacher forcing) that contribute to performance gains.

Abstract

This paper targets a new scenario that integrates speech separation with speech compression, aiming to disentangle multiple speakers while producing discrete representations for efficient transmission or storage, with applications in online meetings and dialogue archiving. To address this scenario, we propose CodeSep, a codec-driven model that jointly performs speech separation and low-bitrate compression. CodeSep comprises a residual vector quantizer (RVQ)-based plain neural speech codec, a base-token disentanglement (BTD) module, and parallel auxiliary-token serial prediction (ATSP) modules. The BTD module disentangles mixed-speech mel-spectrograms into base tokens for each speaker, which are then refined by ATSP modules to serially predict auxiliary tokens, and finally, all tokens are decoded to reconstruct separated waveforms through the codec decoder. During training, the codec's RVQ provides supervision with permutation-invariant and teacher-forcing-based cross-entropy losses. As only base tokens are transmitted or stored, CodeSep achieves low-bitrate compression. Experimental results show that CodeSep attains satisfactory separation performance at only 1 kbps compared with baseline methods.

Figures (1)

• Figure 1: An overview of CodeSep with $N=4$ as an example. The gray parts indicate modules absent during inference.