U3-xi: Pushing the Boundaries of Speaker Recognition via Incorporating Uncertainty

TL;DR

This paper tackles uncertainty in speaker verification by explicitly modeling frame-level data uncertainty and integrating it into both training and scoring. It introduces U3-xi, a framework comprising Stochastic Variance Loss for utterance-level supervision, global-level uncertainty supervision via uncertainty-aware softmax, and a Transformer-based uncertainty estimation module with multi-view self-attention, plus uncertainty-aware scoring. Collectively, these components produce substantial improvements across encoders (e.g., ECAPA-TDNN) and datasets, with reported relative gains such as 21.1% in EER and 15.57% in minDCF on VoxCeleb1 against a strong baseline, while also demonstrating improved interpretability of uncertainty estimates. The work advances robustness and explainability in ASV, though cross-domain calibration remains challenging and highlights the need for further domain adaptation of uncertainty estimates.

Abstract

An utterance-level speaker embedding is typically obtained by aggregating a sequence of frame-level representations. However, in real-world scenarios, individual frames encode not only speaker-relevant information but also various nuisance factors. As a result, different frames contribute unequally to the final utterance-level speaker representation for Automatic Speaker Verification systems. To address this issue, we propose to estimate the inherent uncertainty of each frame and assign adaptive weights accordingly, where frames with higher uncertainty receive lower attention. Based on this idea, we present U3-xi, a comprehensive framework designed to produce more reliable and interpretable uncertainty estimates for speaker embeddings. Specifically, we introduce several strategies for uncertainty supervision. First, we propose speaker-level uncertainty supervision via a Stochastic Variance Loss, where the distance between an utterance embedding and its corresponding speaker centroid serves as a pseudo ground truth for uncertainty learning. Second, we incorporate global-level uncertainty supervision by injecting the predicted uncertainty into the sof tmax scale during training. This adaptive scaling mechanism adjusts the sharpness of the decision boundary according to sample difficulty, providing global guidance. Third, we redesign the uncertainty estimation module by integrating a Transformer encoder with multi-view self-attention, enabling the model to capture rich local and long-range temporal dependencies. Comprehensive experiments demonstrate that U3-xi is model-agnostic and can be seamlessly applied to various speaker encoders. In particular, when applied to ECAPA-TDNN, it achieves 21.1% and 15.57% relative improvements on the VoxCeleb1 test sets in terms of EER and minDCF, respectively.

Figures (12)

• Figure 1: Architecture of the xi-vector model with integrated uncertainty estimation. The mean and variance branches share the same parameters but apply different operations, reflecting the distinct mathematical formulations of mean and variance from a statistical perspective.
• Figure 2: The value difference of the second multiplicative term.
• Figure 3: Illustration of gradient-driven scale adaptation.
• Figure 4: The proposed uncertainty estimation module. Temporal relationships are modeled by a Transformer encoder, and each attention head is assigned with a distinct temporal window, offering multi-view perspectives for estimating frame-level uncertainty.
• Figure 5: Comparison of frame-wise precision under (a) noise corruption and (b) spectrogram masking conditions.