Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Lightweight speech enhancement guided target speech extraction in noisy multi-speaker scenarios

Ziling Huang, Junnan Wu, Lichun Fan, Zhenbo Luo, Jian Luan, Haixin Guan, Yanhua Long

Abstract

Target speech extraction (TSE) has achieved strong performance in relatively simple conditions such as one-speaker-plus-noise and two-speaker mixtures, but its performance remains unsatisfactory in noisy multi-speaker scenarios. To address this issue, we introduce a lightweight speech enhancement model, GTCRN, to better guide TSE in noisy environments. Building on our competitive previous speaker embedding/encoder-free framework SEF-PNet, we propose two extensions: LGTSE and D-LGTSE. LGTSE incorporates noise-agnostic enrollment guidance by denoising the input noisy speech before context interaction with enrollment speech, thereby reducing noise interference. D-LGTSE further improves system robustness against speech distortion by leveraging denoised speech as an additional noisy input during training, expanding the dynamic range of noisy conditions and enabling the model to directly learn from distorted signals. Furthermore, we propose a two-stage training strategy, first with GTCRN enhancement-guided pre-training and then joint fine-tuning, to fully exploit model potential.Experiments on the Libri2Mix dataset demonstrate significant improvements of 0.89 dB in SISDR, 0.16 in PESQ, and 1.97% in STOI, validating the effectiveness of our approach.

Lightweight speech enhancement guided target speech extraction in noisy multi-speaker scenarios

Abstract

Target speech extraction (TSE) has achieved strong performance in relatively simple conditions such as one-speaker-plus-noise and two-speaker mixtures, but its performance remains unsatisfactory in noisy multi-speaker scenarios. To address this issue, we introduce a lightweight speech enhancement model, GTCRN, to better guide TSE in noisy environments. Building on our competitive previous speaker embedding/encoder-free framework SEF-PNet, we propose two extensions: LGTSE and D-LGTSE. LGTSE incorporates noise-agnostic enrollment guidance by denoising the input noisy speech before context interaction with enrollment speech, thereby reducing noise interference. D-LGTSE further improves system robustness against speech distortion by leveraging denoised speech as an additional noisy input during training, expanding the dynamic range of noisy conditions and enabling the model to directly learn from distorted signals. Furthermore, we propose a two-stage training strategy, first with GTCRN enhancement-guided pre-training and then joint fine-tuning, to fully exploit model potential.Experiments on the Libri2Mix dataset demonstrate significant improvements of 0.89 dB in SISDR, 0.16 in PESQ, and 1.97% in STOI, validating the effectiveness of our approach.

Paper Structure

This paper contains 15 sections, 6 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Proposed Methods
  3. Architecture
  4. Noise-agnostic Enrollment Guidance Extraction
  5. Distortion-aware LGTSE (D-LGTSE)
  6. Two-stage Training Strategy
  7. Experiments and Results
  8. Datasets
  9. Models
  10. Configurations
  11. Results
  12. Overall Results
  13. Ablation Study of Model Training Strategy
  14. Visualization of Noise-agnostic Enrollment Guidance
  15. Conclusion

Figures (2)

  • Figure 1: Architecture of the proposed LGTSE and D-LGTSE, and the simplified SEF-PNet (baseline).
  • Figure 2: Noise-agnostic enrollment guidance analysis. The top row shows the enrollment guidance from direct context interaction between enrollment and noisy speech, while the bottom row shows the resulting spectrogram with the proposed noise-agonistic enrollment guidance.