Improving fairness in speaker verification via Group-adapted Fusion Network

TL;DR

This work tackles fairness in speaker verification under demographic imbalance by introducing the Group-adapted Fusion Network (GFN), a modular architecture that performs group-specific embedding adaptation followed by learnable score fusion. The fused score combines cosine similarities from base, female-adapted, and male-adapted embeddings, enabling better generalization and reduced disparity across gender groups. Empirical results on VoxCeleb data show substantial relative reductions in overall $EER$ (9.6%–29.0%), minority-group $EER$ (13.7%–18.6%), and $EER$ disparity (20.0%–25.4%), with notable gains in minority-group performance and visualization-backed evidence of more discriminative, group-aware embeddings. The method is extensible to other skewed training data scenarios and highlights the value of modular, ensemble-like adaptations in fairness-critical acoustic tasks, while also signaling potential future work on robust backends and overfitting safeguards.$S^B$, $S^F$, $S^M$ denote cosine similarities from base and group-adapted embeddings, fused via a learnable $f(ig[S^B,S^F,S^Mig];W)$ to produce $S$.

Abstract

Modern speaker verification models use deep neural networks to encode utterance audio into discriminative embedding vectors. During the training process, these networks are typically optimized to differentiate arbitrary speakers. This learning process biases the learning of fine voice characteristics towards dominant demographic groups, which can lead to an unfair performance disparity across different groups. This is observed especially with underrepresented demographic groups sharing similar voice characteristics. In this work, we investigate the fairness of speaker verification models on controlled datasets with imbalanced gender distributions, providing direct evidence that model performance suffers for underrepresented groups. To mitigate this disparity we propose the group-adapted fusion network (GFN) architecture, a modular architecture based on group embedding adaptation and score fusion. We show that our method alleviates model unfairness by improving speaker verification both overall and for individual groups. Given imbalanced group representation in training, our proposed method achieves overall equal error rate (EER) reduction of 9.6% to 29.0% relative, reduces minority group EER by 13.7% to 18.6%, and results in 20.0% to 25.4% less EER disparity, compared to baselines. The approach is applicable to other types of training data skew in speaker recognition systems.

Figures (4)

• Figure 1: The overview of proposed group-adapted fusion network, which consists of the front-end group adaptation encoders (left) to extract group-wise embeddings and the back-end score fusion model (right) to fuse scores from all embedding encoders.
• Figure 2: VoxCeleb1-F evaluation results from models trained on VoxCeleb2-GRC datasets. Q/RN, H/RN: baseline ResNet models; GFN: gender-adapted fusion of networks.
• Figure 3: Visualization of learned speaker embeddings using the t-SNE method. (a) Male and female speaker utterance embeddings from Q/RN model; (b, c) Utterance embeddings of 8 randomly-sampled speakers from Q/RN model (b) and GFN model (c).
• Figure 4: VoxCeleb1-F evaluation results from models trained on VoxCeleb2-GRC datasets under various ablation study conditions. Black dashed lines are results from our GFN models.