Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Profile-Error-Tolerant Target-Speaker Voice Activity Detection

Dongmei Wang, Xiong Xiao, Naoyuki Kanda, Midia Yousefi, Takuya Yoshioka, Jian Wu

TL;DR

This work addresses the sensitivity of Target-Speaker Voice Activity Detection (TS-VAD) to errors in first-pass speaker profiles by introducing Profile-Error-Tolerant TSVAD (PET-TSVAD). PET-TSVAD augments detected speaker profiles with a fixed set of learnable pseudo-speaker profiles and trains on profiles produced by multiple clustering algorithms, mitigating issues from merging or splitting in the initial diarization. The model uses a transformer-based TSVAD backbone, adds PIT with Hungarian matching to handle permutation ambiguity, and is trained in three stages including simulated pre-training and fine-tuning on VoxConverse and DIHARD-I data. Empirical results show consistent DER reductions compared to baseline TSVAD across both datasets, demonstrating improved robustness to profile errors and better diarization performance in diverse scenarios.

Abstract

Target-Speaker Voice Activity Detection (TS-VAD) utilizes a set of speaker profiles alongside an input audio signal to perform speaker diarization. While its superiority over conventional methods has been demonstrated, the method can suffer from errors in speaker profiles, as those profiles are typically obtained by running a traditional clustering-based diarization method over the input signal. This paper proposes an extension to TS-VAD, called Profile-Error-Tolerant TS-VAD (PET-TSVAD), which is robust to such speaker profile errors. This is achieved by employing transformer-based TS-VAD that can handle a variable number of speakers and further introducing a set of additional pseudo-speaker profiles to handle speakers undetected during the first pass diarization. During training, we use speaker profiles estimated by multiple different clustering algorithms to reduce the mismatch between the training and testing conditions regarding speaker profiles. Experimental results show that PET-TSVAD consistently outperforms the existing TS-VAD method on both the VoxConverse and DIHARD-I datasets.

Profile-Error-Tolerant Target-Speaker Voice Activity Detection

TL;DR

This work addresses the sensitivity of Target-Speaker Voice Activity Detection (TS-VAD) to errors in first-pass speaker profiles by introducing Profile-Error-Tolerant TSVAD (PET-TSVAD). PET-TSVAD augments detected speaker profiles with a fixed set of learnable pseudo-speaker profiles and trains on profiles produced by multiple clustering algorithms, mitigating issues from merging or splitting in the initial diarization. The model uses a transformer-based TSVAD backbone, adds PIT with Hungarian matching to handle permutation ambiguity, and is trained in three stages including simulated pre-training and fine-tuning on VoxConverse and DIHARD-I data. Empirical results show consistent DER reductions compared to baseline TSVAD across both datasets, demonstrating improved robustness to profile errors and better diarization performance in diverse scenarios.

Abstract

Target-Speaker Voice Activity Detection (TS-VAD) utilizes a set of speaker profiles alongside an input audio signal to perform speaker diarization. While its superiority over conventional methods has been demonstrated, the method can suffer from errors in speaker profiles, as those profiles are typically obtained by running a traditional clustering-based diarization method over the input signal. This paper proposes an extension to TS-VAD, called Profile-Error-Tolerant TS-VAD (PET-TSVAD), which is robust to such speaker profile errors. This is achieved by employing transformer-based TS-VAD that can handle a variable number of speakers and further introducing a set of additional pseudo-speaker profiles to handle speakers undetected during the first pass diarization. During training, we use speaker profiles estimated by multiple different clustering algorithms to reduce the mismatch between the training and testing conditions regarding speaker profiles. Experimental results show that PET-TSVAD consistently outperforms the existing TS-VAD method on both the VoxConverse and DIHARD-I datasets.
Paper Structure (19 sections, 1 figure, 4 tables)

This paper contains 19 sections, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Transformer-based TS-VAD: Review
  3. PET-TSVAD
  4. Motivation
  5. Model architecture
  6. Speaker profile generation for training
  7. Model training
  8. Experiments
  9. Evaluation data
  10. Training data and model configuration
  11. Simulated pre-training data
  12. Fine-tuning data
  13. Model configuration
  14. Inference settings
  15. Results
  16. ...and 4 more sections

Figures (1)

  • Figure 1: Model structure of PET-TSVAD.