Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Unmixing the Crowd: Learning Mixture-to-Set Speaker Embeddings for Enrollment-Free Target Speech Extraction

FNU Sidharth, Meysam Asgari, Hao-Wen Dong, Dhruv Jain

Abstract

Personalized or target speech extraction (TSE) typically needs a clean enrollment -- hard to obtain in real-world crowded environments. We remove the essential need for enrollment by predicting, from the mixture itself, a small set of per-speaker embeddings that serve as the control signal for extraction. Our model maps a noisy mixture directly to a small set of candidate speaker embeddings trained to align with a strong single-speaker speaker-embedding space via permutation-invariant teacher supervision. On noisy LibriMix, the resulting embeddings form a structured and clusterable identity space, outperforming WavLM+K-means and separation-derived embeddings in standard clustering metrics. Conditioning these embeddings into multiple extraction back-ends consistently improves objective quality and intelligibility, and generalizes to real DNS-Challenge recordings.

Unmixing the Crowd: Learning Mixture-to-Set Speaker Embeddings for Enrollment-Free Target Speech Extraction

Abstract

Personalized or target speech extraction (TSE) typically needs a clean enrollment -- hard to obtain in real-world crowded environments. We remove the essential need for enrollment by predicting, from the mixture itself, a small set of per-speaker embeddings that serve as the control signal for extraction. Our model maps a noisy mixture directly to a small set of candidate speaker embeddings trained to align with a strong single-speaker speaker-embedding space via permutation-invariant teacher supervision. On noisy LibriMix, the resulting embeddings form a structured and clusterable identity space, outperforming WavLM+K-means and separation-derived embeddings in standard clustering metrics. Conditioning these embeddings into multiple extraction back-ends consistently improves objective quality and intelligibility, and generalizes to real DNS-Challenge recordings.

Paper Structure

This paper contains 16 sections, 2 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. System / Method
  3. Task Definition
  4. Single- and Multi-Speaker Encoders
  5. Experimental Setup
  6. Training and Testing Data
  7. Downstream TSE Task
  8. Optimization and Evaluation
  9. Results
  10. Embedding quality
  11. TSE with mixture-derived embeddings
  12. Cross-dataset evaluation on DNS Challenge (real recordings)
  13. Embedding sensitivity and controllability
  14. Comparison with uPIT separation and separate-then-embed
  15. Conclusion
  16. ...and 1 more sections

Figures (2)

  • Figure 1: Teacher–student framework for mixture-derived multi-speaker embeddings. The teacher defines a single-speaker identity space; the student predicts an unordered set of embeddings from the mixture and is trained via permutation-invariant distillation to stay aligned to the same manifold, encouraging head-wise speaker disentanglement.
  • Figure 2: (a) TSE sensitivity to embedding interpolation/drift (DPCCN) and (b) clustering degradation under separation artifacts.