USED: Universal Speaker Extraction and Diarization

TL;DR

The paper tackles the problem of jointly extracting a target speaker’s speech and diarizing ‘who spoke when’ from mixtures with varying numbers of speakers and arbitrary overlap. It introduces USED, a unified time-domain model that uses an embedding assignment module with active/blank/residual states to produce outputs for a flexible number of speakers, and a multi-task interaction module with a scenario-aware differentiated loss to bridge extraction and diarization. Empirical results on LibriMix, SparseLibriMix, and CALLHOME show significant improvements over strong baselines in both speaker extraction and diarization, with reduced data requirements for real-recordings and competitive complexity. The approach advances practical multi-talker processing by delivering consistent outputs across tasks and robust performance across scenarios, enabling end-to-end ‘who spoke what and when’ systems with a single model.

Abstract

Speaker extraction and diarization are two enabling techniques for real-world speech applications. Speaker extraction aims to extract a target speaker's voice from a speech mixture, while speaker diarization demarcates speech segments by speaker, annotating `who spoke when'. Previous studies have typically treated the two tasks independently. In practical applications, it is more meaningful to have knowledge about `who spoke what and when', which is captured by the two tasks. The two tasks share a similar objective of disentangling speakers. Speaker extraction operates in the frequency domain, whereas diarization is in the temporal domain. It is logical to believe that speaker activities obtained from speaker diarization can benefit speaker extraction, while the extracted speech offers more accurate speaker activity detection than the speech mixture. In this paper, we propose a unified model called Universal Speaker Extraction and Diarization (USED) to address output inconsistency and scenario mismatch issues. It is designed to manage speech mixtures with varying overlap ratios and variable number of speakers. We show that the USED model significantly outperforms the competitive baselines for speaker extraction and diarization tasks on LibriMix and SparseLibriMix datasets. We further validate the diarization performance on CALLHOME, a dataset based on real recordings, and experimental results indicate that our model surpasses recently proposed approaches.

Figures (5)

• Figure 1: The overall training and inference framework of our proposed USED model. It comprises a speech encoder, a speaker encoder, an embedding assignment module, a separator, an extraction decoder and a diarization decoder. The USED model generates both the extracted speech and speaker diarization results by leveraging the speech mixture and the speech references as input. The symbols $\otimes$ and $\ominus$ refer to element-wise multiplication and frame-wise concatenation, respectively. The TCN block and mask module are denoted by rounded rectangles, which are described in detail in Fig. \ref{['fig.modules']}and introduced later. Different colours are used to distinguish network layers from different components. The active, blank and residual states are assigned by Algorithm \ref{['alg:1']}.
• Figure 2: Model structure for a ResNet block and a TCN layer. BN, PReLU, gLN and D-Conv are batch normalization, parametric ReLU, global layer normalization and dilated depth-wise separable convolution. The symbol $\oplus$ refers to element-wise addition.
• Figure 3: Assigning States for Embeddings
• Figure 4: Illustration of scenario-aware differentiated loss. The speech mixture is segmented according to the four scenarios (QQ, QS, SS and SQ).
• Figure 5: Performance on the SparseLibriMix for overlap ratio from 0% to 100%. The Left and right bar charts are the results of speaker diarization and speaker extraction tasks, respectively.