Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Leveraging Language Information for Target Language Extraction

Mehmet Sinan Yıldırım, Ruijie Tao, Wupeng Wang, Junyi Ao, Haizhou Li

TL;DR

Target Language Extraction (TLE) aims to isolate speech in a chosen language from multilingual mixtures. We propose injecting language knowledge by using a self-supervised multilingual pre-trained model to provide an auxiliary loss that guides the extraction, implemented in a two-stage training regime. A new public dataset, CommonVoiceMix, enables fair benchmarking of English–German TLE; the method achieves SI-SNR improvements of $1.22$ dB for English and $1.12$ dB for German, with STOI and PESQ gains and no inference cost increase. This work demonstrates that language-aware supervision can boost TLE performance and provides a reusable baseline and dataset to drive future multilingual TLE research.

Abstract

Target Language Extraction aims to extract speech in a specific language from a mixture waveform that contains multiple speakers speaking different languages. The human auditory system is adept at performing this task with the knowledge of the particular language. However, the performance of the conventional extraction systems is limited by the lack of this prior knowledge. Speech pre-trained models, which capture rich linguistic and phonetic representations from large-scale in-the-wild corpora, can provide this missing language knowledge to these systems. In this work, we propose a novel end-to-end framework to leverage language knowledge from speech pre-trained models. This knowledge is used to guide the extraction model to better capture the target language characteristics, thereby improving extraction quality. To demonstrate the effectiveness of our proposed approach, we construct the first publicly available multilingual dataset for Target Language Extraction. Experimental results show that our method achieves improvements of 1.22 dB and 1.12 dB in SI-SNR for English and German extraction, respectively, from mixtures containing both languages.

Leveraging Language Information for Target Language Extraction

TL;DR

Target Language Extraction (TLE) aims to isolate speech in a chosen language from multilingual mixtures. We propose injecting language knowledge by using a self-supervised multilingual pre-trained model to provide an auxiliary loss that guides the extraction, implemented in a two-stage training regime. A new public dataset, CommonVoiceMix, enables fair benchmarking of English–German TLE; the method achieves SI-SNR improvements of dB for English and dB for German, with STOI and PESQ gains and no inference cost increase. This work demonstrates that language-aware supervision can boost TLE performance and provides a reusable baseline and dataset to drive future multilingual TLE research.

Abstract

Target Language Extraction aims to extract speech in a specific language from a mixture waveform that contains multiple speakers speaking different languages. The human auditory system is adept at performing this task with the knowledge of the particular language. However, the performance of the conventional extraction systems is limited by the lack of this prior knowledge. Speech pre-trained models, which capture rich linguistic and phonetic representations from large-scale in-the-wild corpora, can provide this missing language knowledge to these systems. In this work, we propose a novel end-to-end framework to leverage language knowledge from speech pre-trained models. This knowledge is used to guide the extraction model to better capture the target language characteristics, thereby improving extraction quality. To demonstrate the effectiveness of our proposed approach, we construct the first publicly available multilingual dataset for Target Language Extraction. Experimental results show that our method achieves improvements of 1.22 dB and 1.12 dB in SI-SNR for English and German extraction, respectively, from mixtures containing both languages.

Paper Structure

This paper contains 14 sections, 6 equations, 2 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Target Language Extraction with Language Information
  3. Target Language Extraction
  4. Proposed Method for Injecting Language Information
  5. CommonVoiceMix: A New Public Corpus for TLE
  6. Experimental Setup
  7. Experiments and Results
  8. Main results
  9. Ablation studies
  10. Using HuBERT
  11. Different $\beta$ values
  12. Smaller model size
  13. Different pre-trained model
  14. Conclusion

Figures (2)

  • Figure 1: Overview of the Target Language Extraction (TLE) task: given a speech mixture containing multiple languages (e.g. English and German), the model extracts only the speech corresponding to the target language (e.g., English), regardless of speaker identity.
  • Figure 2: Overview of the proposed method. Pre-trained speech model obtains embeddings for both the estimated and the target waveforms. An additional similarity loss between them contributes to align the internal speech representations of the base model