Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Detect, Attend and Extract: Keyword Guided Target Speaker Extraction

Haoyu Li, Yu Xi, Yidi Jiang, Shuai Wang, Kate Knill, Mark Gales, Haizhou Li, Kai Yu

TL;DR

DAE-TSE is proposed, a keyword-guided TSE framework that specifies the target speaker through distinct keywords they utter, using partial transcription as a cue for specifying the target speaker in TSE, offering a flexible and practical solution for real-world scenarios.

Abstract

Target speaker extraction (TSE) aims to extract the speech of a target speaker from mixtures containing multiple competing speakers. Conventional TSE systems predominantly rely on speaker cues, such as pre-enrolled speech, to identify and isolate the target speaker. However, in many practical scenarios, clean enrollment utterances are unavailable, limiting the applicability of existing approaches. In this work, we propose DAE-TSE, a keyword-guided TSE framework that specifies the target speaker through distinct keywords they utter. By leveraging keywords (i.e., partial transcriptions) as cues, our approach provides a flexible and practical alternative to enrollment-based TSE. DAE-TSE follows the Detect-Attend-Extract (DAE) paradigm: it first detects the presence of the given keywords, then attends to the corresponding speaker based on the keyword content, and finally extracts the target speech. Experimental results demonstrate that DAE-TSE outperforms standard TSE systems that rely on clean enrollment speech. To the best of our knowledge, this is the first study to utilize partial transcription as a cue for specifying the target speaker in TSE, offering a flexible and practical solution for real-world scenarios. Our code and demo page are now publicly available.

Detect, Attend and Extract: Keyword Guided Target Speaker Extraction

TL;DR

DAE-TSE is proposed, a keyword-guided TSE framework that specifies the target speaker through distinct keywords they utter, using partial transcription as a cue for specifying the target speaker in TSE, offering a flexible and practical solution for real-world scenarios.

Abstract

Target speaker extraction (TSE) aims to extract the speech of a target speaker from mixtures containing multiple competing speakers. Conventional TSE systems predominantly rely on speaker cues, such as pre-enrolled speech, to identify and isolate the target speaker. However, in many practical scenarios, clean enrollment utterances are unavailable, limiting the applicability of existing approaches. In this work, we propose DAE-TSE, a keyword-guided TSE framework that specifies the target speaker through distinct keywords they utter. By leveraging keywords (i.e., partial transcriptions) as cues, our approach provides a flexible and practical alternative to enrollment-based TSE. DAE-TSE follows the Detect-Attend-Extract (DAE) paradigm: it first detects the presence of the given keywords, then attends to the corresponding speaker based on the keyword content, and finally extracts the target speech. Experimental results demonstrate that DAE-TSE outperforms standard TSE systems that rely on clean enrollment speech. To the best of our knowledge, this is the first study to utilize partial transcription as a cue for specifying the target speaker in TSE, offering a flexible and practical solution for real-world scenarios. Our code and demo page are now publicly available.
Paper Structure (25 sections, 17 equations, 4 figures, 4 tables, 1 algorithm)

This paper contains 25 sections, 17 equations, 4 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. DAE-TSE: Keyword-Guided Target Speaker Extraction
  3. Target Speaker Extraction Fundamentals
  4. Keyword-guided Cue Encoder for DAE-TSE
  5. Architecture
  6. Training Criteria
  7. Speech Extraction Backbone for DAE-TSE
  8. DAE: Detect, Attend and Extract
  9. Experimental Setups
  10. Data Preparation for KCE module
  11. Data Simulation.
  12. Keyword Selection
  13. Evaluation Dataset Simulation
  14. Data Preparation for Extraction Backbone
  15. Training Details
  16. ...and 10 more sections

Figures (4)

  • Figure 1: An illustration of the application scenario and objectives of the proposed DAE-TSE framework. In multi-talker scenarios, DAE-TSE aims to extract the speech of the target speaker who uttered the given keywords, such as "Hey Siri," from the mixture.
  • Figure 2: The overview of the proposed DAE-TSE. (1) The left part of the figure is the structure of the Keyword-guided Cue Encoder (KCE). KCE is trained by both ASR loss and SV loss. The inputs of KCE are a mixture speech and keywords, and then output the target speaker cue embedding. (2) The right part of the figure is the TSE training process with pretrained KCE, where the inputs are a mixture speech and keywords, and then output the extracted target speech.
  • Figure 3: Cross-attention heatmaps for a positive sample (left, keywords present in the mixture) and a negative sample (right, keywords absent). The X-axis represents speech frame indices, and the Y-axis corresponds to the phoneme sequence of the keyword.
  • Figure 4: t-SNE scatter plot of speaker embeddings, with each color representing a target speaker. (a) Embeddings in the same color are extracted from the same mixture across different keywords. (b) Embeddings in the same color are extracted from different mixtures.