Self-Distillation Prototypes Network: Learning Robust Speaker Representations without Supervision

TL;DR

SDPN tackles unsupervised speaker verification by learning robust embeddings without speaker labels. It introduces a teacher–student framework with shared learnable prototypes and a multi-crop strategy to align global and local views, optimized by a combined objective $L = L_{CE} + \mu L_{DR}$. Prototypes encode implicit speaker classes, while the diversity regularization term mitigates model collapse, enabling strong discriminability in a non-contrastive SSL setting. Experiments on VoxCeleb demonstrate state-of-the-art performance on VoxCeleb1 in all trials without labeled data, with significant gains attributed to the prototypes and the diversity term.

Abstract

Training speaker-discriminative and robust speaker verification systems without explicit speaker labels remains a persistent challenge. In this paper, we propose a novel self-supervised speaker verification approach, Self-Distillation Prototypes Network (SDPN), which effectively facilitates self-supervised speaker representation learning. SDPN assigns the representation of the augmented views of an utterance to the same prototypes as the representation of the original view, thereby enabling effective knowledge transfer between the augmented and original views. Due to lack of negative pairs in the SDPN training process, the network tends to align positive pairs quite closely in the embedding space, a phenomenon known as model collapse. To mitigate this problem, we introduce a diversity regularization term to embeddings in SDPN. Comprehensive experiments on the VoxCeleb datasets demonstrate the superiority of SDPN among self-supervised speaker verification approaches. SDPN sets a new state-of-the-art on the VoxCeleb1 speaker verification evaluation benchmark, achieving Equal Error Rate 1.80%, 1.99%, and 3.62% for trial VoxCeleb1-O, VoxCeleb1-E and VoxCeleb1-H, without using any speaker labels in training. Ablation studies show that both proposed learnable prototypes in self-distillation network and diversity regularization contribute to the verification performance.

Figures (2)

• Figure 1: Overview of our proposed Self-Distillation Prototypes Network (SDPN). SDPN comprises a teacher network and a student network with identical architecture but different parameters. The teacher/student network consists of three main modules: the encoder for extracting speaker embeddings, a multi-layer perceptron for feature transformation, and learnable prototypes for computing soft distributions between global and local views. Global views and local views refer to the long and short segments randomly segmented from the same utterance, respectively. EMA denotes Exponential Moving Average.
• Figure 2: t-SNE visualization of extracted embeddings for five speakers, each denoted by a distinct color. The left figure displays speaker embeddings from baseline DINO, while the right figure shows those from our SDPN with diversity regularization. The embeddings from SDPN with diversity regularization clearly exhibit enhanced separation over those from DINO, suggesting better discriminability.