Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TS-SUPERB: A Target Speech Processing Benchmark for Speech Self-Supervised Learning Models

Junyi Peng, Takanori Ashihara, Marc Delcroix, Tsubasa Ochiai, Oldrich Plchot, Shoko Araki, Jan Černocký

TL;DR

TS-SUPERB introduces a standardized benchmark for target-speaker speech processing, extending SSL evaluation beyond single-speaker tasks to four downstream tasks: TSE, PSE, PVAD, and TS-ASR. It uses a unified target speech encoder, conditioned on enrollment speech, with task-specific decoders, and explores multi-task learning to leverage shared information across tasks. Across seven SSL models and two dataset regimes (Libri2Mix and Noisy SparseLibri2Mix), the results show that no single model dominates all TS tasks; models with denoising augmentations (e.g., WavLM variants) are particularly strong in denoising-related tasks, while TS-ASR benefits most from larger models. The findings highlight the unique challenges of target-speaker processing and demonstrate that joint training can yield mutual benefits, motivating broader adoption of TS-SUPERB to push SSL methods toward robust, multi-task, target-aware speech processing. $L = \alpha L_i + (1-\alpha) L_j$ captures the core multi-task objective framework used to jointly optimize task pairs.$

Abstract

Self-supervised learning (SSL) models have significantly advanced speech processing tasks, and several benchmarks have been proposed to validate their effectiveness. However, previous benchmarks have primarily focused on single-speaker scenarios, with less exploration of target-speaker tasks in noisy, multi-talker conditions -- a more challenging yet practical case. In this paper, we introduce the Target-Speaker Speech Processing Universal Performance Benchmark (TS-SUPERB), which includes four widely recognized target-speaker processing tasks that require identifying the target speaker and extracting information from the speech mixture. In our benchmark, the speaker embedding extracted from enrollment speech is used as a clue to condition downstream models. The benchmark result reveals the importance of evaluating SSL models in target speaker scenarios, demonstrating that performance cannot be easily inferred from related single-speaker tasks. Moreover, by using a unified SSL-based target speech encoder, consisting of a speaker encoder and an extractor module, we also investigate joint optimization across TS tasks to leverage mutual information and demonstrate its effectiveness.

TS-SUPERB: A Target Speech Processing Benchmark for Speech Self-Supervised Learning Models

TL;DR

TS-SUPERB introduces a standardized benchmark for target-speaker speech processing, extending SSL evaluation beyond single-speaker tasks to four downstream tasks: TSE, PSE, PVAD, and TS-ASR. It uses a unified target speech encoder, conditioned on enrollment speech, with task-specific decoders, and explores multi-task learning to leverage shared information across tasks. Across seven SSL models and two dataset regimes (Libri2Mix and Noisy SparseLibri2Mix), the results show that no single model dominates all TS tasks; models with denoising augmentations (e.g., WavLM variants) are particularly strong in denoising-related tasks, while TS-ASR benefits most from larger models. The findings highlight the unique challenges of target-speaker processing and demonstrate that joint training can yield mutual benefits, motivating broader adoption of TS-SUPERB to push SSL methods toward robust, multi-task, target-aware speech processing. captures the core multi-task objective framework used to jointly optimize task pairs.$

Abstract

Self-supervised learning (SSL) models have significantly advanced speech processing tasks, and several benchmarks have been proposed to validate their effectiveness. However, previous benchmarks have primarily focused on single-speaker scenarios, with less exploration of target-speaker tasks in noisy, multi-talker conditions -- a more challenging yet practical case. In this paper, we introduce the Target-Speaker Speech Processing Universal Performance Benchmark (TS-SUPERB), which includes four widely recognized target-speaker processing tasks that require identifying the target speaker and extracting information from the speech mixture. In our benchmark, the speaker embedding extracted from enrollment speech is used as a clue to condition downstream models. The benchmark result reveals the importance of evaluating SSL models in target speaker scenarios, demonstrating that performance cannot be easily inferred from related single-speaker tasks. Moreover, by using a unified SSL-based target speech encoder, consisting of a speaker encoder and an extractor module, we also investigate joint optimization across TS tasks to leverage mutual information and demonstrate its effectiveness.
Paper Structure (16 sections, 3 figures, 8 tables)

This paper contains 16 sections, 3 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. TS-SUPERB
  4. Datasets
  5. Target Speech Encoder
  6. Target Speech Extraction (TSE)
  7. Personalized Speech Enhancement (PSE)
  8. Personalized VAD (PVAD)
  9. Target-speaker ASR (TS-ASR)
  10. Multi-task Learning
  11. Results and Analysis
  12. Performance comparison of various SSL models
  13. Layer-wise and task correlation analysis
  14. Analysis of multi-task learning and fine-tuning
  15. Conclusion
  16. ...and 1 more sections

Figures (3)

  • Figure 1: Architecture of proposed TS-SUPERB system. All downstream models use the same architecture for the target speaker encoder, followed by the TSE, PSE, TS-ASR, or PVAD decoder, depending on the task.
  • Figure 2: The weight distribution of SSL layers. Note that the 0th layer denotes the input of the 1st Transformer encoder layer.
  • Figure 3: The Spearman's rank correlation between tasks.