Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Benchmarking Diarization Models

Luca A. Lanzendörfer, Florian Grötschla, Cesare Blaser, Roger Wattenhofer

TL;DR

This study benchmarks five diarization models across four multilingual datasets, spanning meetings, phone calls, and in-the-wild audio, to determine practical deployment guidance and failure modes. It compares modular, end-to-end, streaming, and hybrid approaches, including Pyannote, PyannoteAI, Sortformer variants, and DiariZen. PyannoteAI achieves the lowest overall DER (11.2%), with DiariZen close behind (13.3%), while Sortformer v2 streaming stands out for speed (RTF ≈214x) and robust performance in certain conditions. Across languages and speaker counts, data availability emerges as a key driver of performance, with missed speech and timing precision identified as the main failure modes, suggesting future work should prioritize onset/end-point accuracy for improvements in diarization systems.

Abstract

Speaker diarization is the task of partitioning audio into segments according to speaker identity, answering the question of "who spoke when" in multi-speaker conversation recordings. While diarization is an essential task for many downstream applications, it remains an unsolved problem. Errors in diarization propagate to downstream systems and cause wide-ranging failures. To this end, we examine exact failure modes by evaluating five state-of-the-art diarization models, across four diarization datasets spanning multiple languages and acoustic conditions. The evaluation datasets consist of 196.6 hours of multilingual audio, including English, Mandarin, German, Japanese, and Spanish. Overall, we find that PyannoteAI achieves the best performance at 11.2% DER, while DiariZen provides a competitive open-source alternative at 13.3% DER. When analyzing failure cases, we find that the primary cause of diarization errors stem from missed speech segments followed by speaker confusion, especially in high-speaker count settings.

Benchmarking Diarization Models

TL;DR

This study benchmarks five diarization models across four multilingual datasets, spanning meetings, phone calls, and in-the-wild audio, to determine practical deployment guidance and failure modes. It compares modular, end-to-end, streaming, and hybrid approaches, including Pyannote, PyannoteAI, Sortformer variants, and DiariZen. PyannoteAI achieves the lowest overall DER (11.2%), with DiariZen close behind (13.3%), while Sortformer v2 streaming stands out for speed (RTF ≈214x) and robust performance in certain conditions. Across languages and speaker counts, data availability emerges as a key driver of performance, with missed speech and timing precision identified as the main failure modes, suggesting future work should prioritize onset/end-point accuracy for improvements in diarization systems.

Abstract

Speaker diarization is the task of partitioning audio into segments according to speaker identity, answering the question of "who spoke when" in multi-speaker conversation recordings. While diarization is an essential task for many downstream applications, it remains an unsolved problem. Errors in diarization propagate to downstream systems and cause wide-ranging failures. To this end, we examine exact failure modes by evaluating five state-of-the-art diarization models, across four diarization datasets spanning multiple languages and acoustic conditions. The evaluation datasets consist of 196.6 hours of multilingual audio, including English, Mandarin, German, Japanese, and Spanish. Overall, we find that PyannoteAI achieves the best performance at 11.2% DER, while DiariZen provides a competitive open-source alternative at 13.3% DER. When analyzing failure cases, we find that the primary cause of diarization errors stem from missed speech segments followed by speaker confusion, especially in high-speaker count settings.

Paper Structure

This paper contains 10 sections, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Models
  4. Model Adaptations & Evaluation Metric
  5. Datasets
  6. Results
  7. General Performance
  8. Per-Language Performance
  9. Per-Speaker Count Performance
  10. Conclusion

Figures (2)

  • Figure 1: Diarization error rate across models showing missed speech (red), false alarm (orange), and speaker confusion (blue) components. We find that Sortformer v2-streaming, DiariZen, and the commercial PyannoteAI models perform the best overall. The numbers above the bar represent the total DER (%, lower is better) for each model. We find that missed speech errors are the most common source of errors, with the exception of Sortformer, which has a higher speaker confusion error than other diarization models.
  • Figure 2: We visualize the distribution of speech segment lengths found over all datasets (left). For each diarization model we visualize in which segment length missed speech occurred (right). We find that models exhibit mostly unbiased error patterns across segment lengths, matching dataset composition, indicating that the failures are not from missing short segments but rather from inaccurate onset and end timestamp information across all segment length types.