Improving Speaker Representations Using Contrastive Losses on Multi-scale Features

TL;DR

This work proposes a Multi-scale Feature Contrastive (MFCon) loss that applies contrastive learning to all feature maps within the network, encouraging the model to learn more discriminative representations at the intermediate stage itself.

Abstract

Speaker verification systems have seen significant advancements with the introduction of Multi-scale Feature Aggregation (MFA) architectures, such as MFA-Conformer and ECAPA-TDNN. These models leverage information from various network depths by concatenating intermediate feature maps before the pooling and projection layers, demonstrating that even shallower feature maps encode valuable speaker-specific information. Building upon this foundation, we propose a Multi-scale Feature Contrastive (MFCon) loss that directly enhances the quality of these intermediate representations. Our MFCon loss applies contrastive learning to all feature maps within the network, encouraging the model to learn more discriminative representations at the intermediate stage itself. By enforcing better feature map learning, we show that the resulting speaker embeddings exhibit increased discriminative power. Our method achieves a 9.05% improvement in equal error rate (EER) compared to the standard MFA-Conformer on the VoxCeleb-1O test set.

Figures (2)

• Figure 1: The MFCon loss applied to the MFA conformer model. The arrows in black show the original MFA conformer and the arrows in red show the architecture to obtain embeddings from the feature maps of the Conformer blocks
• Figure 2: Illustration of the MFCon idea. Utterances x1 and x2 from distinct speakers, along with their augmentations y1 and y2, are processed through N conformer blocks. Feature map embeddings are extracted at each scale using dedicated encoders. The MFCon loss applies contrastive learning across all scales, simultaneously minimizing intra-speaker distances and maximizing inter-speaker distances in the embedding space