Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Joint Training of Speaker Embedding Extractor, Speech and Overlap Detection for Diarization

Petr Pálka, Federico Landini, Dominik Klement, Mireia Diez, Anna Silnova, Marc Delcroix, Lukáš Burget

TL;DR

This work proposes an approach to jointly train a model to produce speaker embeddings, VAD and OSD simultaneously and reach competitive performance at a fraction of the inference time of a modular approach.

Abstract

In spite of the popularity of end-to-end diarization systems nowadays, modular systems comprised of voice activity detection (VAD), speaker embedding extraction plus clustering, and overlapped speech detection (OSD) plus handling still attain competitive performance in many conditions. However, one of the main drawbacks of modular systems is the need to run (and train) different modules independently. In this work, we propose an approach to jointly train a model to produce speaker embeddings, VAD and OSD simultaneously and reach competitive performance at a fraction of the inference time of a standard approach. Furthermore, the joint inference leads to a simplified overall pipeline which brings us one step closer to a unified clustering-based method that can be trained end-to-end towards a diarization-specific objective.

Joint Training of Speaker Embedding Extractor, Speech and Overlap Detection for Diarization

TL;DR

This work proposes an approach to jointly train a model to produce speaker embeddings, VAD and OSD simultaneously and reach competitive performance at a fraction of the inference time of a modular approach.

Abstract

In spite of the popularity of end-to-end diarization systems nowadays, modular systems comprised of voice activity detection (VAD), speaker embedding extraction plus clustering, and overlapped speech detection (OSD) plus handling still attain competitive performance in many conditions. However, one of the main drawbacks of modular systems is the need to run (and train) different modules independently. In this work, we propose an approach to jointly train a model to produce speaker embeddings, VAD and OSD simultaneously and reach competitive performance at a fraction of the inference time of a standard approach. Furthermore, the joint inference leads to a simplified overall pipeline which brings us one step closer to a unified clustering-based method that can be trained end-to-end towards a diarization-specific objective.

Paper Structure

This paper contains 14 sections, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Standard Methods
  3. Diarization system pipeline
  4. Baseline embedding extraction
  5. Proposed Modifications
  6. High-resolution embedding extraction
  7. Integrated VAD and OSD
  8. Experimental Setup
  9. Datasets
  10. Baseline and proposed method configurations
  11. Results
  12. Per-frame embeddings
  13. Joint training
  14. Conclusions

Figures (2)

  • Figure 1: Comparison of modular diarization pipelines.
  • Figure 2: Standard and proposed embedding extraction. In the proposed approach, all embeddings are used to calculate the VAD loss, but only those corresponding to speech are used for the OSD loss. Speaker embeddings are denoted in cyan.