Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Attractor-Based Speech Separation of Multiple Utterances by Unknown Number of Speakers

Yuzhu Wang, Archontis Politis, Konstantinos Drossos, Tuomas Virtanen

TL;DR

Addresses single-channel speech separation with an unknown number of speakers $C$, where each speaker may contribute multiple utterances. Proposes an attractor-based joint diarization, counting, and separation system (A-DCSS) that uses an LSTM-based attractor module to generate $J$ attractors plus a non-existence attractor, FiLM conditioning, dual-path transformer embeddings, and a triple-path separator; the model optimizes a joint loss $\mathcal{L}_{\text{joint}} = \lambda_{\text{s}} \mathcal{L}_{\text{SI-SDR}} + \lambda_{\text{d}} \mathcal{L}_{\text{diar}} + \lambda_{\text{e}} \mathcal{L}_{\text{exist}}$. Datasets synthesize LibriSpeech with WHAM! noise across anechoic, noisy, reverberant, and noisy+reverberant conditions, training in two phases. Findings show A-DCSS accurately estimates source count and speaker activity, improves separation over baselines in both fixed and unknown speaker-count scenarios, and remains robust to noise and reverberation.

Abstract

This paper addresses the problem of single-channel speech separation, where the number of speakers is unknown, and each speaker may speak multiple utterances. We propose a speech separation model that simultaneously performs separation, dynamically estimates the number of speakers, and detects individual speaker activities by integrating an attractor module. The proposed system outperforms existing methods by introducing an attractor-based architecture that effectively combines local and global temporal modeling for multi-utterance scenarios. To evaluate the method in reverberant and noisy conditions, a multi-speaker multi-utterance dataset was synthesized by combining Librispeech speech signals with WHAM! noise signals. The results demonstrate that the proposed system accurately estimates the number of sources. The system effectively detects source activities and separates the corresponding utterances into correct outputs in both known and unknown source count scenarios.

Attractor-Based Speech Separation of Multiple Utterances by Unknown Number of Speakers

TL;DR

Addresses single-channel speech separation with an unknown number of speakers , where each speaker may contribute multiple utterances. Proposes an attractor-based joint diarization, counting, and separation system (A-DCSS) that uses an LSTM-based attractor module to generate attractors plus a non-existence attractor, FiLM conditioning, dual-path transformer embeddings, and a triple-path separator; the model optimizes a joint loss . Datasets synthesize LibriSpeech with WHAM! noise across anechoic, noisy, reverberant, and noisy+reverberant conditions, training in two phases. Findings show A-DCSS accurately estimates source count and speaker activity, improves separation over baselines in both fixed and unknown speaker-count scenarios, and remains robust to noise and reverberation.

Abstract

This paper addresses the problem of single-channel speech separation, where the number of speakers is unknown, and each speaker may speak multiple utterances. We propose a speech separation model that simultaneously performs separation, dynamically estimates the number of speakers, and detects individual speaker activities by integrating an attractor module. The proposed system outperforms existing methods by introducing an attractor-based architecture that effectively combines local and global temporal modeling for multi-utterance scenarios. To evaluate the method in reverberant and noisy conditions, a multi-speaker multi-utterance dataset was synthesized by combining Librispeech speech signals with WHAM! noise signals. The results demonstrate that the proposed system accurately estimates the number of sources. The system effectively detects source activities and separates the corresponding utterances into correct outputs in both known and unknown source count scenarios.

Paper Structure

This paper contains 12 sections, 2 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Signal Model
  3. Attractor-based joint diarization, counting, and separation system (A-DCSS)
  4. Experimental settings
  5. Datasets
  6. Training Configurations
  7. Evaluation Metrics
  8. Experimental Results
  9. Fixed Number of Speakers
  10. Unknown Number of Speakers
  11. Ablation experiment
  12. Conclusions

Figures (4)

  • Figure 1: The illustration of the A-DCSS in multi-speaker with multi-utterance scenarios.
  • Figure 2: A-DCSS architecture.
  • Figure 3: Dual-path (dashed line) and triple-path modules. $\mathbf{D}_{\text{out}}$ and $\mathbf{T}_{\text{out}}$ are their outputs with inputs $\mathbf{D}_{\text{in}}$ and $\mathbf{T}_{\text{in}}$, respectively. $\text{P}$ and $\text{R}$ represent the permute and reshape operations.
  • Figure 4: Attractor module.