Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Speaker Contrastive Learning for Source Speaker Tracing

Qing Wang, Hongmei Guo, Jian Kang, Mengjie Du, Jie Li, Xiao-Lei Zhang, Lei Xie

TL;DR

The paper tackles the vulnerability of speaker verification to voice conversion by introducing the Source Speaker Tracing Challenge (SSTC) and proposing a speaker contrastive learning framework to extract latent source-speaker information from converted speech. A three-phase training regime coupled with a joint loss $\mathcal{L} = \mathcal{L}_{\text{AAM}} + \alpha \mathcal{L}_{\text{Con}}$ and a contrastive term $\mathcal{L}_{\text{Con}}$ using $K=5$ negatives and temperature $\tau$ enables the embedding extractor to identify the true source speaker among distractors. Empirically, the method achieves $EER = 16.788\%$ on the SSTC test set, ranking first and outperforming the baseline by up to $3.825\%$ absolute, demonstrating robust extraction of source-speaker traces across diverse VC methods. The approach advances practical source speaker tracing under VC by leveraging latent source-speaker information retained in converted speech.

Abstract

As a form of biometric authentication technology, the security of speaker verification systems is of utmost importance. However, SV systems are inherently vulnerable to various types of attacks that can compromise their accuracy and reliability. One such attack is voice conversion, which modifies a persons speech to sound like another person by altering various vocal characteristics. This poses a significant threat to SV systems. To address this challenge, the Source Speaker Tracing Challenge in IEEE SLT2024 aims to identify the source speaker information in manipulated speech signals. Specifically, SSTC focuses on source speaker verification against voice conversion to determine whether two converted speech samples originate from the same source speaker. In this study, we propose a speaker contrastive learning-based approach for source speaker tracing to learn the latent source speaker information in converted speech. To learn a more source-speaker-related representation, we employ speaker contrastive loss during the training of the embedding extractor. This speaker contrastive loss helps identify the true source speaker embedding among several distractor speaker embeddings, enabling the embedding extractor to learn the potentially possessing source speaker information present in the converted speech. Experiments demonstrate that our proposed speaker contrastive learning system achieves the lowest EER of 16.788% on the challenge test set, securing first place in the challenge.

Speaker Contrastive Learning for Source Speaker Tracing

TL;DR

The paper tackles the vulnerability of speaker verification to voice conversion by introducing the Source Speaker Tracing Challenge (SSTC) and proposing a speaker contrastive learning framework to extract latent source-speaker information from converted speech. A three-phase training regime coupled with a joint loss and a contrastive term using negatives and temperature enables the embedding extractor to identify the true source speaker among distractors. Empirically, the method achieves on the SSTC test set, ranking first and outperforming the baseline by up to absolute, demonstrating robust extraction of source-speaker traces across diverse VC methods. The approach advances practical source speaker tracing under VC by leveraging latent source-speaker information retained in converted speech.

Abstract

As a form of biometric authentication technology, the security of speaker verification systems is of utmost importance. However, SV systems are inherently vulnerable to various types of attacks that can compromise their accuracy and reliability. One such attack is voice conversion, which modifies a persons speech to sound like another person by altering various vocal characteristics. This poses a significant threat to SV systems. To address this challenge, the Source Speaker Tracing Challenge in IEEE SLT2024 aims to identify the source speaker information in manipulated speech signals. Specifically, SSTC focuses on source speaker verification against voice conversion to determine whether two converted speech samples originate from the same source speaker. In this study, we propose a speaker contrastive learning-based approach for source speaker tracing to learn the latent source speaker information in converted speech. To learn a more source-speaker-related representation, we employ speaker contrastive loss during the training of the embedding extractor. This speaker contrastive loss helps identify the true source speaker embedding among several distractor speaker embeddings, enabling the embedding extractor to learn the potentially possessing source speaker information present in the converted speech. Experiments demonstrate that our proposed speaker contrastive learning system achieves the lowest EER of 16.788% on the challenge test set, securing first place in the challenge.
Paper Structure (10 sections, 2 equations, 2 figures, 5 tables)

This paper contains 10 sections, 2 equations, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Speaker Embedding Clustering
  4. Adapter-based Multi-Task Learning
  5. Speaker Contrastive Learning
  6. EXPERIMENTAL SETUP AND RESULTS
  7. Dataset
  8. Setup
  9. Evaluation Results
  10. CONCLUSION

Figures (2)

  • Figure 1: Source speaker verification against voice conversion.
  • Figure 2: Overview of the proposed source speaker contrastive learning system. $\mathbf{e}_c$ and $E_s$ refer to the representation of the embedding extraction layer and a set of embeddings of source speech, respectively. Here, $+$ and $-$ represent positive and negative sample pairs, respectively, which are used to compute the speaker contrastive loss $\mathcal{L}_{\text{Con}}$.