Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SLM-SS: Speech Language Model for Generative Speech Separation

Tianhua Li, Chenda Li, Wei Wang, Xin Zhou, Xihui Chen, Jianqing Gao, Yanmin Qian

TL;DR

SLM-SS introduces a Speech Language Model-based framework for speech separation by discretizing speech into multi-codebook sequences via Encodec and assembling them with Serialized Output Training. It employs a hybrid decoding strategy—an autoregressive AED for zero-order codebooks followed by a non-autoregressive (NAR) model for higher-order codebooks—to generate coherent, intelligible separated speech. Experiments on LibriMix show that SLM-SS improves speech intelligibility and downstream linguistic metrics (WER, LPS, SBS) relative to baselines, with Encodec-based discretization maintaining perceptual quality. The work highlights the potential of integrating SS with language-model-like decoding and suggests future directions toward unified SS-ASR systems that operate on discrete tokens.

Abstract

Speech separation (SS) has advanced significantly with neural network-based methods, showing improved performance on signal-level metrics. However, these methods often struggle to maintain speech intelligibility in the separated signals, which can negatively affect the performance of downstream tasks such as speech recognition. In this work, we propose SLM-SS, a novel approach that applies speech language models to SS, aiming to enhance the intelligibility and coherence of the separated signals. We frame SS as discrete multi-codebook sequence generation, using Encoder-Decoder models to map quantized speech mixtures to target tokens. In addition to the autoregressive modeling strategy, we introduce a non-autoregressive model to improve decoding efficiency for residual tokens. Experimental results on the LibriMix dataset demonstrate that our approach shows significantly better preservation of speech intelligibility, leading to improved linguistic consistency in a variety of downstream tasks compared to existing approaches.

SLM-SS: Speech Language Model for Generative Speech Separation

TL;DR

SLM-SS introduces a Speech Language Model-based framework for speech separation by discretizing speech into multi-codebook sequences via Encodec and assembling them with Serialized Output Training. It employs a hybrid decoding strategy—an autoregressive AED for zero-order codebooks followed by a non-autoregressive (NAR) model for higher-order codebooks—to generate coherent, intelligible separated speech. Experiments on LibriMix show that SLM-SS improves speech intelligibility and downstream linguistic metrics (WER, LPS, SBS) relative to baselines, with Encodec-based discretization maintaining perceptual quality. The work highlights the potential of integrating SS with language-model-like decoding and suggests future directions toward unified SS-ASR systems that operate on discrete tokens.

Abstract

Speech separation (SS) has advanced significantly with neural network-based methods, showing improved performance on signal-level metrics. However, these methods often struggle to maintain speech intelligibility in the separated signals, which can negatively affect the performance of downstream tasks such as speech recognition. In this work, we propose SLM-SS, a novel approach that applies speech language models to SS, aiming to enhance the intelligibility and coherence of the separated signals. We frame SS as discrete multi-codebook sequence generation, using Encoder-Decoder models to map quantized speech mixtures to target tokens. In addition to the autoregressive modeling strategy, we introduce a non-autoregressive model to improve decoding efficiency for residual tokens. Experimental results on the LibriMix dataset demonstrate that our approach shows significantly better preservation of speech intelligibility, leading to improved linguistic consistency in a variety of downstream tasks compared to existing approaches.
Paper Structure (12 sections, 5 equations, 2 figures, 2 tables)

This paper contains 12 sections, 5 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Generative Modeling of Speech Separation
  4. Hybrid Decoding for Progressive Codec Generation
  5. AED Model for Zero-order Codebook Sequence
  6. NAR Model for High-order Codebook Sequences
  7. Experiment
  8. Experiment Setup
  9. Results Analysis
  10. Ablation on number of codebooks
  11. Ablation on temperature during infernece
  12. Conclusion

Figures (2)

  • Figure 1: Overview of SLM-SS. (a) Encodec quantizes single-speaker audio into multi-codebook sequences then merges them using SOT. (b) AED model predicts the zero-order codebook sequence. (c) NAR model sequentially predicts higher-order codebook sequences given lower-order ones. (d) SOT sequences are segmented into single-person sequences then decoded into audio. (e) NAR decoder employs multiple independent token embeddings to integrate all low-order sequence information.
  • Figure 2: Variation of WER and LPS with Codebooks.