Incorporating Spatial Cues in Modular Speaker Diarization for Multi-channel Multi-party Meetings

TL;DR

The paper addresses diarization in multi-channel, multi-party meetings by proposing a three-stage modular system that leverages spatial cues to progressively improve NSD initializations. Stage 1 separates overlapping speech with multi-channel CSS and initializes clustering; Stage 2 uses these results to initialize a cACGMM over multi-channel data and derive refined VAD per speaker; Stage 3 applies GSS with short-segment filtering and re-clustering to finalize NSD. Across CHiME-8 NOTSOFAR-1 scenarios, the approach yields clearer speaker boundaries and improved recognition (DER and tcpWER), with Stage 2 often delivering the strongest diarization gains and Stage 3 optimizing recognition by suppressing ultra-short segments. This work demonstrates that integrating spatial processing into modular NSD pipelines enhances robustness and performance in realistic far-field, multi-party settings, achieving first place in the challenge and offering practical benefits for downstream ASR and meeting analytics.

Abstract

Although fully end-to-end speaker diarization systems have made significant progress in recent years, modular systems often achieve superior results in real-world scenarios due to their greater adaptability and robustness. Historically, modular speaker diarization methods have seldom discussed how to leverage spatial cues from multi-channel speech. This paper proposes a three-stage modular system to enhance single-channel neural speaker diarization systems and recognition performance by utilizing spatial cues from multi-channel speech to provide more accurate initialization for each stage of neural speaker diarization (NSD) decoding: (1) Overlap detection and continuous speech separation (CSS) on multi-channel speech are used to obtain cleaner single speaker speech segments for clustering, followed by the first NSD decoding pass. (2) The results from the first pass initialize a complex Angular Central Gaussian Mixture Model (cACGMM) to estimate speaker-wise masks on multi-channel speech, and through Overlap-add and Mask-to-VAD, achieve initialization with lower speaker error (SpkErr), followed by the second NSD decoding pass. (3) The second decoding results are used for guided source separation (GSS), recognizing and filtering short segments containing less one word to obtain cleaner speech segments, followed by re-clustering and the final NSD decoding pass. We presented the progressively explored evaluation results from the CHiME-8 NOTSOFAR-1 (Natural Office Talkers in Settings Of Far-field Audio Recordings) challenge, demonstrating the effectiveness of our system and its contribution to improving recognition performance. Our final system achieved the first place in the challenge.

Figures (1)

• Figure 1: The pipeline for the proposed three-stage modular speaker diarization.