Noise-Conditioned Mixture-of-Experts Framework for Robust Speaker Verification

TL;DR

This work tackles robust speaker verification in noisy environments by introducing a noise-conditioned mixture-of-experts (NCMoE) framework that partitions the feature space into noise-aware subspaces. A lightweight noise classifier routes each input to a single specialized expert, while a universal-model based pretraining followed by noise-conditioned specialization (UMES) and an SNR-decaying curriculum (SDCL) enhance stability and generalization. Empirical results on VoxCeleb1 with MUSAN and Nonspeech100 demonstrate consistent improvements over baselines and prior methods, including strong cross-domain generalization to unseen noise. The study highlights the value of explicit, noise-dependent feature modeling for robust SV without sacrificing Speaker Verification accuracy, with avenues for richer mixtures and broader noise categories in future work.

Abstract

Robust speaker verification under noisy conditions remains an open challenge. Conventional deep learning methods learn a robust unified speaker representation space against diverse background noise and achieve significant improvement. In contrast, this paper presents a noise-conditioned mixture-ofexperts framework that decomposes the feature space into specialized noise-aware subspaces for speaker verification. Specifically, we propose a noise-conditioned expert routing mechanism, a universal model based expert specialization strategy, and an SNR-decaying curriculum learning protocol, collectively improving model robustness and generalization under diverse noise conditions. The proposed method can automatically route inputs to expert networks based on noise information derived from the inputs, where each expert targets distinct noise characteristics while preserving speaker identity information. Comprehensive experiments demonstrate consistent superiority over baselines, confirming that explicit noise-dependent feature modeling significantly enhances robustness without sacrificing verification accuracy.

Figures (1)

• Figure 1: The noise-conditioned mixture-of-expert framework.