Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Listen to Extract: Onset-Prompted Target Speaker Extraction

Pengjie Shen, Kangrui Chen, Shulin He, Pengru Chen, Shuqi Yuan, He Kong, Xueliang Zhang, Zhong-Qiu Wang

TL;DR

This work addresses the problem of monaural target speaker extraction (TSE) by proposing Listen to Extract (LExt), a simple yet effective onset-prompted approach. By prepending a target speaker's enrollment to the mixture, LExt creates an artificial onset that guides the network to extract that speaker without introducing extra conditioning modules. The method demonstrates strong performance on WSJ0-2mix, WHAM!, and WHAMR! datasets, often surpassing state-of-the-art embedding-based TSE methods while using standard architectures like TF-GridNet and TF-LocoFormer. The study analyzes design choices such as enrollment length, glue signals, and the trade-offs between prepending versus appending, and discusses practical limitations for offline and real-time deployment, highlighting LExt’s potential to influence future TSE design.

Abstract

We propose listen to extract (LExt), a highly-effective while extremely-simple algorithm for monaural target speaker extraction (TSE). Given an enrollment utterance of a target speaker, LExt aims at extracting the target speaker from the speaker's mixed speech with other speakers. For each mixture, LExt concatenates an enrollment utterance of the target speaker to the mixture signal at the waveform level, and trains deep neural networks (DNN) to extract the target speech based on the concatenated mixture signal. The rationale is that, this way, an artificial speech onset is created for the target speaker and it could prompt the DNN (a) which speaker is the target to extract; and (b) spectral-temporal patterns of the target speaker that could help extraction. This simple approach produces strong TSE performance on multiple public TSE datasets including WSJ0-2mix, WHAM! and WHAMR!.

Listen to Extract: Onset-Prompted Target Speaker Extraction

TL;DR

This work addresses the problem of monaural target speaker extraction (TSE) by proposing Listen to Extract (LExt), a simple yet effective onset-prompted approach. By prepending a target speaker's enrollment to the mixture, LExt creates an artificial onset that guides the network to extract that speaker without introducing extra conditioning modules. The method demonstrates strong performance on WSJ0-2mix, WHAM!, and WHAMR! datasets, often surpassing state-of-the-art embedding-based TSE methods while using standard architectures like TF-GridNet and TF-LocoFormer. The study analyzes design choices such as enrollment length, glue signals, and the trade-offs between prepending versus appending, and discusses practical limitations for offline and real-time deployment, highlighting LExt’s potential to influence future TSE design.

Abstract

We propose listen to extract (LExt), a highly-effective while extremely-simple algorithm for monaural target speaker extraction (TSE). Given an enrollment utterance of a target speaker, LExt aims at extracting the target speaker from the speaker's mixed speech with other speakers. For each mixture, LExt concatenates an enrollment utterance of the target speaker to the mixture signal at the waveform level, and trains deep neural networks (DNN) to extract the target speech based on the concatenated mixture signal. The rationale is that, this way, an artificial speech onset is created for the target speaker and it could prompt the DNN (a) which speaker is the target to extract; and (b) spectral-temporal patterns of the target speaker that could help extraction. This simple approach produces strong TSE performance on multiple public TSE datasets including WSJ0-2mix, WHAM! and WHAMR!.
Paper Structure (33 sections, 2 equations, 3 figures, 6 tables)

This paper contains 33 sections, 2 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. TSE Methods Not Using Fixed-Length Speaker Embeddings
  4. Leveraging Estimated Onset for TSE
  5. Onset-Prompted Overlapped Speech Processing
  6. LExt
  7. Listen to Extract (LExt)
  8. Length and Values of Glue Signal
  9. Loss Functions
  10. DNN Architectures
  11. Gain Normalization of Enrollment Utterance and Mixture
  12. Only Prepend, or Not Only Prepend But Also Append?
  13. Removing Silences in Enrollment Utterance
  14. Experimental Setup
  15. Datasets
  16. ...and 18 more sections

Figures (3)

  • Figure 1: Illustration of LExt for TSE. In this example, an enrollment utterance and a glue signal are prepended to the mixture for the DNN to extract the target speaker. Best viewed in color. The mixture signal (shown in black) consists of the target speech (in red) and non-target signals (in blue).
  • Figure 2: Histogram of SI-SDRi scores on WSJ0-2mix test set. "Inf" denotes infinity.
  • Figure 3: Illustration of attention maps in TFGridNetV1 based LExt reported in row 1C of Table \ref{['results_enrollment_length']}. In the top-left sub-plot, we mark down the time ranges of the enrollment utterance, glue signal and mixture. Best viewed in color.