End-to-End Integration of Speech Separation and Voice Activity Detection for Low-Latency Diarization of Telephone Conversations

TL;DR

The paper tackles low-latency diarization in conversational telephone speech by integrating speech separation and voice activity detection into a fully end-to-end SSGD framework, augmented with a lightweight leakage removal module. It evaluates multiple causal SSep models (Conv-TasNet, DPTNet, DPRNN) and VAD variants on Fisher and CALLHOME, including end-to-end fine-tuning strategies that bypass the need for oracle separated sources. The end-to-end SSGD with SSep+VAD fine-tuning achieves state-of-the-art or surpasses online EEND performance on 2-speaker CALLHOME with a latency of only 0.1 s, while maintaining competitive separation metrics. The framework enables downstream ASR with substantial gains over mixtures, though end-to-end diarization can introduce distortions that affect WER compared to oracle sources. The work provides thorough online/offline analyses, CSS window Trade-offs, and real-time feasibility, offering concrete directions for extending to more speakers and hybrid online clustering approaches.

Abstract

Recent works show that speech separation guided diarization (SSGD) is an increasingly promising direction, mainly thanks to the recent progress in speech separation. It performs diarization by first separating the speakers and then applying voice activity detection (VAD) on each separated stream. In this work we conduct an in-depth study of SSGD in the conversational telephone speech (CTS) domain, focusing mainly on low-latency streaming diarization applications. We consider three state-of-the-art speech separation (SSep) algorithms and study their performance both in online and offline scenarios, considering non-causal and causal implementations as well as continuous SSep (CSS) windowed inference. We compare different SSGD algorithms on two widely used CTS datasets: CALLHOME and Fisher Corpus (Part 1 and 2) and evaluate both separation and diarization performance. To improve performance, a novel, causal and computationally efficient leakage removal algorithm is proposed, which significantly decreases false alarms. We also explore, for the first time, fully end-to-end SSGD integration between SSep and VAD modules. Crucially, this enables fine-tuning on real-world data for which oracle speakers sources are not available. In particular, our best model achieves 8.8% DER on CALLHOME, which outperforms the current state-of-the-art end-to-end neural diarization model, despite being trained on an order of magnitude less data and having significantly lower latency, i.e., 0.1 vs. 1 s. Finally, we also show that the separated signals can be readily used also for automatic speech recognition, reaching performance close to using oracle sources in some configurations.

Figures (7)

• Figure 1: General diagram of the SSGD method.
• Figure 2: Separation and diarization results on the test sets with different CSS windows. The overlap between windows is set to $50$%. The results are obtained with the disjoint SSGD model (DPRNN+TCN+Leakage removal).
• Figure 3: Separation and diarization results on the test sets with the online DPTNet-based SSGD by varying latency.
• Figure 4: Diarization results of the disjoint SSGD with and without leakage removal on the test sets by varying VAD threshold.
• Figure 5: Diarization results of the end-to-end SSGD with and without leakage removal on the test sets by varying VAD threshold.