Self-Supervised Learning for Speaker Recognition: A study and review

TL;DR

This work surveys and analyzes self-supervised learning for speaker verification, adapting CV SSL paradigms—contrastive learning, clustering, information maximization, and self-distillation—to learn speaker representations from unlabeled data. It provides a unified experimental protocol and a comparative study across single-stage and multi-stage SV methods, detailing how hyperparameters (negatives, temperature, momentum, frame sampling) and components (data augmentation, projector, positive sampling) affect performance. Key findings show DINO as a strong single-stage method that excels with larger encoders and carefully tuned training setups, while SimCLR, MoCo, SwAV, and VICReg offer robust alternatives; multi-stage approaches with pseudo-label refinement push state-of-the-art on VoxCeleb benchmarks. The work also demonstrates SSL’s label efficiency, achieving competitive results with substantially fewer labeled samples, and outlines practical directions—scaling data, exploring encoder capacity, and leveraging multi-modal cues—for advancing SV with SSL.

Abstract

Deep learning models trained in a supervised setting have revolutionized audio and speech processing. However, their performance inherently depends on the quantity of human-annotated data, making them costly to scale and prone to poor generalization under unseen conditions. To address these challenges, Self-Supervised Learning (SSL) has emerged as a promising paradigm, leveraging vast amounts of unlabeled data to learn relevant representations. The application of SSL for Automatic Speech Recognition (ASR) has been extensively studied, but research on other downstream tasks, notably Speaker Recognition (SR), remains in its early stages. This work describes major SSL instance-invariance frameworks (e.g., SimCLR, MoCo, and DINO), initially developed for computer vision, along with their adaptation to SR. Various SSL methods for SR, proposed in the literature and built upon these frameworks, are also presented. An extensive review of these approaches is then conducted: (1) the effect of the main hyperparameters of SSL frameworks is investigated; (2) the role of SSL components is studied (e.g., data-augmentation, projector, positive sampling); and (3) SSL frameworks are evaluated on SR with in-domain and out-of-domain data, using a consistent experimental setup, and a comprehensive comparison of SSL methods from the literature is provided. Specifically, DINO achieves the best downstream performance and effectively models intra-speaker variability, although it is highly sensitive to hyperparameters and training conditions, while SimCLR and MoCo provide robust alternatives that effectively capture inter-speaker variability and are less prone to collapse. This work aims to highlight recent trends and advancements, identifying current challenges in the field.

Figures (10)

• Figure 1: SSL framework for SV. The training (a) is performed on a pretext task to learn relevant representations, which will be used to perform the evaluation (b) on the downstream task. The training framework adopts the joint-embedding architecture to generate a pair of embeddings (anchor and positive) from an unlabeled audio waveform.
• Figure 2: Conceptual comparison of different SSL frameworks across various paradigms. The joint-embedding architecture remains constant, but the mechanisms to prevent collapse vary between methods. Encoders are represented in green, projectors in red, predictors in purple, operations in yellow, and losses in blue. Rectangular-shaped modules process fixed-length feature vectors or sequences of feature vectors, while trapezoid-shaped modules, functioning as encoders or autoregressive models, reduce variable-length sequences into fixed-length feature vectors. F-norm refers to feature normalization, B-norm to batch normalization, EMA to Exponential Moving Average, and sg to stop-gradient.
• Figure 3: Timeline of a selection of single-stage SSL methods for SV from the literature. Methods are categorized by framework: Contrastive learning or Self-distillation. The release date is determined according to the conference or journal publication date of the corresponding article.
• Figure 4: Collapse study of MoCo (a) and DINO (b) SSL frameworks. For MoCo (a), the entropy of the contrastive distribution and the standard deviation of embeddings are reported. For DINO (b), the entropy of the teacher distribution and the KL divergence of the teacher and student distributions are reported. Metrics are reported throughout training iterations of the first 3 epochs. The encoder is Fast ResNet-34.
• Figure 5: Performance of SSL frameworks on SV with different probabilities of applying data-augmentation. The encoder is Fast ResNet-34 and the EER is reported on VoxCeleb1-O.