USEF-TSE: Universal Speaker Embedding Free Target Speaker Extraction

TL;DR

This work tackles target speaker extraction without relying on speaker embeddings by introducing USEF-TSE, a universal embedding-free framework that can operate in both time-domain and time-frequency domains. It leverages a cross multi-head attention module to derive a frame-level target speaker feature from enrollment speech and fuses it with the mixed speech encoding before separation, enabling seamless integration with backbones like SepFormer and TF-GridNet. The proposed USEF-SepFormer and USEF-TFGridNet achieve state-of-the-art SI-SDRi on WSJ0-2mix, WHAM!, WHAMR!, LibriMix, and show robust generalization on DNS Challenge data, closely approaching or surpassing embedding-based baselines. This framework reduces dependence on speaker recognizers, improves utilization of enrollment context, and broadens practical applicability of TSE across noisy, reverberant, and out-of-domain conditions.

Abstract

Target speaker extraction aims to separate the voice of a specific speaker from mixed speech. Traditionally, this process has relied on extracting a speaker embedding from a reference speech, in which a speaker recognition model is required. However, identifying an appropriate speaker recognition model can be challenging, and using the target speaker embedding as reference information may not be optimal for target speaker extraction tasks. This paper introduces a Universal Speaker Embedding-Free Target Speaker Extraction (USEF-TSE) framework that operates without relying on speaker embeddings. USEF-TSE utilizes a multi-head cross-attention mechanism as a frame-level target speaker feature extractor. This innovative approach allows mainstream speaker extraction solutions to bypass the dependency on speaker recognition models and better leverage the information available in the enrollment speech, including speaker characteristics and contextual details. Additionally, USEF-TSE can seamlessly integrate with other time-domain or time-frequency domain speech separation models to achieve effective speaker extraction. Experimental results show that our proposed method achieves state-of-the-art (SOTA) performance in terms of Scale-Invariant Signal-to-Distortion Ratio (SI-SDR) on the WSJ0-2mix, WHAM!, and WHAMR! datasets, which are standard benchmarks for monaural anechoic, noisy and noisy-reverberant two-speaker speech separation and speaker extraction. The results on the LibriMix and the blind test set of the ICASSP 2023 DNS Challenge demonstrate that the model performs well on more diverse and out-of-domain data. For access to the source code, please visit: https://github.com/ZBang/USEF-TSE.

Figures (6)

• Figure 1: The diagram of a typical target speaker extraction method. The speaker embedding extractor is typically a pre-trained speaker recognition model. 'C' denotes the concatenation.
• Figure 2: The diagram of the SEF-Net. ‘m’ and ‘r’ denote the mixed speech and reference speech, respectively. ‘$\textbf{E}_{m}$’ and ‘$\textbf{E}_{r}$’ denote the Encoder output of mixed and reference speech, respectively. ‘$\textbf{S}_{m}$’ and ‘$\textbf{S}_{r}$’ denote the segmentation result of ‘$\textbf{E}_{m}$’ and ‘$\textbf{E}_{r}$’, respectively. ‘$\textbf{D}_{\text{intra}}$’ and ‘$\textbf{D}_{\text{inter}}$’ denote the output of the Intra and Inter Module, respectively.
• Figure 3: The diagram of the USEF-TSE framework. ‘m’ and ‘r’ denote the mixed speech and reference speech, respectively. ‘est’ denotes the estimation of the target speaker.
• Figure 4: The diagram of USEF-SepFormer network. ‘m’ and ‘r’ denote the mixed speech and reference speech, respectively. We use two weight sharing encoder to process the mixed and reference speech separately. ‘est’ denotes the estimation of the target speaker. $\otimes$ is an operation for element-wise product. The Separator’s parameters are set identically to those of the SepFormer approach.
• Figure 5: The diagram of USEF-TFGridNet network. We use the same STFT settings to extract the acoustic features of both the mixed and the reference speech. The weight shared Conv2d increase the number of channel for both the mixed and reference speech' acoustic features. ‘m’ and ‘r’ denote the mixed speech and reference speech, respectively. ‘est’ denotes the estimation of the target speaker. The Separator’s parameters are set identically to those of the TF-GridNet approach.