TS-SEP: Joint Diarization and Separation Conditioned on Estimated Speaker Embeddings

TL;DR

This paper addresses the intertwined tasks of diarization and speech separation in meeting data by introducing TS-SEP, an extension of TS-VAD that outputs time-frequency speaker masks to enable joint diarization and source extraction. The method uses an initial speaker embedding estimation stage and a two-stage training regime, culminating in a TF-resolved output that supports masking and beamforming-based extraction. Experimental results on LibriCSS demonstrate state-of-the-art diarization-aware word error rates, with strong performance gains when using WavLM-based ASR and multi-channel inputs, while the framework provides detailed analyses via DIcpWER to separate diarization errors from recognition errors. The approach promises practical impact for end-to-end meeting transcription by tightly integrating diarization with high-quality source extraction, and it opens avenues for applying to real recordings like CHiME-6 and for hybrid training on real data.

Abstract

Since diarization and source separation of meeting data are closely related tasks, we here propose an approach to perform the two objectives jointly. It builds upon the target-speaker voice activity detection (TS-VAD) diarization approach, which assumes that initial speaker embeddings are available. We replace the final combined speaker activity estimation network of TS-VAD with a network that produces speaker activity estimates at a time-frequency resolution. Those act as masks for source extraction, either via masking or via beamforming. The technique can be applied both for single-channel and multi-channel input and, in both cases, achieves a new state-of-the-art word error rate (WER) on the LibriCSS meeting data recognition task. We further compute speaker-aware and speaker-agnostic WERs to isolate the contribution of diarization errors to the overall WER performance.

Figures (6)

• Figure 1: TS-VAD and TS-SEP structure for one microphone. Speakers indicated as "shading".
• Figure 2: System overview: Pretraining with TS-VAD loss, training with TS-SEP loss and predicting transcriptions at inference. Microphones indicated as "shading".
• Figure 3: Estimated speaker activity ${\tilde{A}_{t, k}}$ for 2 of 8 speakers from the LibriCSS DEV OV40 dataset, shown together with the corresponding clean speaker time signals $\boldsymbol{\mathbf{x}}_{\ell, k}$. Best viewed in color.
• Figure 4: Thresholding and “closing” morphological operation (dilation followed by erosion) on ${\tilde{A}_{t, k}}$ to obtain ${\hat{A}_{t, k}}$, for 2 of 8 speakers from the LibriCSS DEV OV40 dataset. Best viewed in color.
• Figure 5: Distribution of the number of speakers active in 1.0-minute chunks of the training data, depending on the probability $\zeta$ to use superposition/mixup ($\zeta=0$ corresponds to the original dataset without superposition).