MK-SGC-SC: Multiple Kernel Guided Sparse Graph Construction in Spectral Clustering for Unsupervised Speaker Diarization

TL;DR

The paper tackles unsupervised speaker diarization by proposing MK-SGC-SC, a method that builds a sparse, cluster-aware affinity graph from multiple kernel similarities of speaker embeddings and then applies spectral clustering. By using four polynomial kernels plus a degree-one arccosine kernel and fusing the resulting graphs, the approach emphasizes local structure and yields strong DER performance on DIHARD-III, AMI, and VoxConverse without supervision. The main contributions are (i) introducing multiple kernel similarities tailored for SD, (ii) a principled fusion and sparsification framework to construct a single adjacency matrix, and (iii) comprehensive experiments showing MK-SGC-SC often outperforms state-of-the-art unsupervised methods and rivals semi-supervised approaches. The work demonstrates that kernel diversification coupled with sparse graph fusion can match or exceed supervised baselines, offering a scalable, unsupervised alternative for speaker diarization in diverse environments.

Abstract

Speaker diarization aims to segment audio recordings into regions corresponding to individual speakers. Although unsupervised speaker diarization is inherently challenging, the prospect of identifying speaker regions without pretraining or weak supervision motivates research on clustering techniques. In this work, we share the notable observation that measuring multiple kernel similarities of speaker embeddings to thereafter craft a sparse graph for spectral clustering in a principled manner is sufficient to achieve state-of-the-art performances in a fully unsupervised setting. Specifically, we consider four polynomial kernels and a degree one arccosine kernel to measure similarities in speaker embeddings, using which sparse graphs are constructed in a principled manner to emphasize local similarities. Experiments show the proposed approach excels in unsupervised speaker diarization over a variety of challenging environments in the DIHARD-III, AMI, and VoxConverse corpora. To encourage further research, our implementations are available at https://github.com/nikhilraghav29/MK-SGC-SC.