Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

3S-TSE: Efficient Three-Stage Target Speaker Extraction for Real-Time and Low-Resource Applications

Shulin He, Jinjiang liu, Hao Li, Yang Yang, Fei Chen, Xueliang Zhang

TL;DR

This paper tackles real-time target speaker extraction under resource constraints by introducing 3S-TSE, a modular three-stage framework that first estimates the target's direction of arrival (DOA) with an encoder-attentive recurrent network, then performs Generalized Sidelobe Canceller (GSC) beamforming guided by the DOA, and finally refines the output with an InplaceCRN post-processing module. The approach achieves strong objective performance while remaining highly compact (about 0.19M parameters) and computationally efficient (approximately 1.000G MACs), demonstrated on LibriSpeech-derived, reverberant six-mic data and yielding a notable STOI improvement of 17.3%. By decoupling DOA estimation, beamforming, and denoising, the method offers a practical path toward real-time, edge-optimized TSE, outperforming end-to-end large models and showing robustness to reverberation. The modular design enables efficient deployment on resource-limited platforms and sets a new standard for low-resource, real-time target speech extraction.

Abstract

Target speaker extraction (TSE) aims to isolate a specific voice from multiple mixed speakers relying on a registerd sample. Since voiceprint features usually vary greatly, current end-to-end neural networks require large model parameters which are computational intensive and impractical for real-time applications, espetially on resource-constrained platforms. In this paper, we address the TSE task using microphone array and introduce a novel three-stage solution that systematically decouples the process: First, a neural network is trained to estimate the direction of the target speaker. Second, with the direction determined, the Generalized Sidelobe Canceller (GSC) is used to extract the target speech. Third, an Inplace Convolutional Recurrent Neural Network (ICRN) acts as a denoising post-processor, refining the GSC output to yield the final separated speech. Our approach delivers superior performance while drastically reducing computational load, setting a new standard for efficient real-time target speaker extraction.

3S-TSE: Efficient Three-Stage Target Speaker Extraction for Real-Time and Low-Resource Applications

TL;DR

This paper tackles real-time target speaker extraction under resource constraints by introducing 3S-TSE, a modular three-stage framework that first estimates the target's direction of arrival (DOA) with an encoder-attentive recurrent network, then performs Generalized Sidelobe Canceller (GSC) beamforming guided by the DOA, and finally refines the output with an InplaceCRN post-processing module. The approach achieves strong objective performance while remaining highly compact (about 0.19M parameters) and computationally efficient (approximately 1.000G MACs), demonstrated on LibriSpeech-derived, reverberant six-mic data and yielding a notable STOI improvement of 17.3%. By decoupling DOA estimation, beamforming, and denoising, the method offers a practical path toward real-time, edge-optimized TSE, outperforming end-to-end large models and showing robustness to reverberation. The modular design enables efficient deployment on resource-limited platforms and sets a new standard for low-resource, real-time target speech extraction.

Abstract

Target speaker extraction (TSE) aims to isolate a specific voice from multiple mixed speakers relying on a registerd sample. Since voiceprint features usually vary greatly, current end-to-end neural networks require large model parameters which are computational intensive and impractical for real-time applications, espetially on resource-constrained platforms. In this paper, we address the TSE task using microphone array and introduce a novel three-stage solution that systematically decouples the process: First, a neural network is trained to estimate the direction of the target speaker. Second, with the direction determined, the Generalized Sidelobe Canceller (GSC) is used to extract the target speech. Third, an Inplace Convolutional Recurrent Neural Network (ICRN) acts as a denoising post-processor, refining the GSC output to yield the final separated speech. Our approach delivers superior performance while drastically reducing computational load, setting a new standard for efficient real-time target speaker extraction.
Paper Structure (12 sections, 4 figures, 4 tables)

This paper contains 12 sections, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Notations
  4. System overview
  5. First Stage: TS-DOA
  6. Second Stage: GSC
  7. Third Stage: The Post-Processing module
  8. Loss function
  9. Experiments
  10. Dataset and setup
  11. Experiment results
  12. CONCLUSIONS

Figures (4)

  • Figure 1: Overview of the proposed 3S-TSE system.
  • Figure 2: Structure of the six-channel microphone array. For brevity, the figure presents the source signal distribution from 0° to 180°. The unseen distribution from 180° to 360° is a mirror reflection of the shown pattern, ensuring symmetry.
  • Figure 3: (First Stage) Block Diagram of TS-DOA Module. Note: an output angle value of -1 indicates the absence of the target speaker's voice in that frame.
  • Figure 4: (Three Stage) Block Diagram of Inplace CRN Post-Processing Module.