MMM: Multi-Layer Multi-Residual Multi-Stream Discrete Speech Representation from Self-supervised Learning Model

TL;DR

MMM targets the efficiency gap of discrete speech representations by introducing multi-stream discrete tokens extracted from self-supervised learning (SSL) models through iterative residual vector quantization (RVQ). It supports both single-layer and multi-layer configurations, enabling richer representations by combining multiple streams across one or more SSL layers. Across ASR, speech resynthesis, and TTS, MMM consistently improves performance over single-stream SSL tokens and matches or surpasses neural codec baselines in several settings, approaching the performance of continuous SSL representations. This work offers a practical path to compact, high-capacity speech representations with broad applicability and improved interoperability across downstream tasks.

Abstract

Speech discrete representation has proven effective in various downstream applications due to its superior compression rate of the waveform, fast convergence during training, and compatibility with other modalities. Discrete units extracted from self-supervised learning (SSL) models have emerged as a prominent approach for obtaining speech discrete representation. However, while discrete units have shown effectiveness compared to spectral features, they still lag behind continuous SSL representations. In this work, we propose MMM, a multi-layer multi-residual multi-stream discrete units extraction method from SSL. Specifically, we introduce iterative residual vector quantization with K-means for different layers in an SSL model to extract multi-stream speech discrete representation. Through extensive experiments in speech recognition, speech resynthesis, and text-to-speech, we demonstrate the proposed MMM can surpass or on-par with neural codec's performance under various conditions.