A Stage-Wise Learning Strategy with Fixed Anchors for Robust Speaker Verification
Bin Gu, Lipeng Dai, Huipeng Du, Haitao Zhao, Jibo Wei
TL;DR
The paper tackles robust speaker verification under noise by introducing a two-stage, anchor-guided learning strategy. It first trains a base model to maximize discriminability, then fixes anchor embeddings derived from clean speech and fine-tunes a copy on noisy inputs using an anchor-driven intra-variance suppression loss that exploits an exponential cosine distance. This separation of boundary stabilization from noise-robust embedding learning preserves inter-speaker separation while reducing intra-speaker variance under distortion, outperforming joint optimization baselines and showing stronger generalization to unseen noise. The approach yields stable decision boundaries and more compact speaker representations, with practical implications for real-world, noise-robust SV systems.
Abstract
Learning robust speaker representations under noisy conditions presents significant challenges, which requires careful handling of both discriminative and noise-invariant properties. In this work, we proposed an anchor-based stage-wise learning strategy for robust speaker representation learning. Specifically, our approach begins by training a base model to establish discriminative speaker boundaries, and then extract anchor embeddings from this model as stable references. Finally, a copy of the base model is fine-tuned on noisy inputs, regularized by enforcing proximity to their corresponding fixed anchor embeddings to preserve speaker identity under distortion. Experimental results suggest that this strategy offers advantages over conventional joint optimization, particularly in maintaining discrimination while improving noise robustness. The proposed method demonstrates consistent improvements across various noise conditions, potentially due to its ability to handle boundary stabilization and variation suppression separately.