Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring wav2vec 2.0 on speaker verification and language identification

Zhiyun Fan, Meng Li, Shiyu Zhou, Bo Xu

TL;DR

This work investigates applying wav2vec 2.0's self-supervised pre-training to speaker verification and language identification. It demonstrates that pre-trained context representations retain speaker and language cues, and fine-tuning yields strong SV performance (3.61% EER on VoxCeleb1) and competitive LID results (12.02% EER on 1s, 3.47% on full-length AP17-OLR). A multi-task fine-tuning approach further enables unified modeling with reduced parameters. Overall, the study shows that self-supervised speech representations can be effectively transferred to SV and LID tasks, with practical benefits for multi-task deployment.

Abstract

Wav2vec 2.0 is a recently proposed self-supervised framework for speech representation learning. It follows a two-stage training process of pre-training and fine-tuning, and performs well in speech recognition tasks especially ultra-low resource cases. In this work, we attempt to extend self-supervised framework to speaker verification and language identification. First, we use some preliminary experiments to indicate that wav2vec 2.0 can capture the information about the speaker and language. Then we demonstrate the effectiveness of wav2vec 2.0 on the two tasks respectively. For speaker verification, we obtain a new state-of-the-art result, Equal Error Rate (EER) of 3.61% on the VoxCeleb1 dataset. For language identification, we obtain an EER of 12.02% on 1 second condition and an EER of 3.47% on full-length condition of the AP17-OLR dataset. Finally, we utilize one model to achieve the unified modeling by the multi-task learning for the two tasks.

Exploring wav2vec 2.0 on speaker verification and language identification

TL;DR

This work investigates applying wav2vec 2.0's self-supervised pre-training to speaker verification and language identification. It demonstrates that pre-trained context representations retain speaker and language cues, and fine-tuning yields strong SV performance (3.61% EER on VoxCeleb1) and competitive LID results (12.02% EER on 1s, 3.47% on full-length AP17-OLR). A multi-task fine-tuning approach further enables unified modeling with reduced parameters. Overall, the study shows that self-supervised speech representations can be effectively transferred to SV and LID tasks, with practical benefits for multi-task deployment.

Abstract

Wav2vec 2.0 is a recently proposed self-supervised framework for speech representation learning. It follows a two-stage training process of pre-training and fine-tuning, and performs well in speech recognition tasks especially ultra-low resource cases. In this work, we attempt to extend self-supervised framework to speaker verification and language identification. First, we use some preliminary experiments to indicate that wav2vec 2.0 can capture the information about the speaker and language. Then we demonstrate the effectiveness of wav2vec 2.0 on the two tasks respectively. For speaker verification, we obtain a new state-of-the-art result, Equal Error Rate (EER) of 3.61% on the VoxCeleb1 dataset. For language identification, we obtain an EER of 12.02% on 1 second condition and an EER of 3.47% on full-length condition of the AP17-OLR dataset. Finally, we utilize one model to achieve the unified modeling by the multi-task learning for the two tasks.

Paper Structure

This paper contains 12 sections, 5 equations, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Pre-training of wav2vec 2.0
  4. Fine-tuning
  5. Experiments
  6. Datasets
  7. Model description
  8. Feasibility analysis
  9. Speaker verification
  10. Language identification
  11. Multi-task system
  12. Conclusion

Figures (2)

  • Figure 1: An overview of the pre-training and fine-tuning. The models architecture used in pre-training stage and fine-tuning stage are identical, except the quantization modules and extra output layers.
  • Figure 2: 2D t-SNE plot of representations extracted from the bottom layer (layer1), the middle layer (layer6), and the high layer (layer12) of the Transformer of M-nofinetune. The left column is the clustering of the representations of 10 speakers in the test set of VoxCeleb1 extracted by M-nofinetune, and each color represents a speaker. The right column is the clustering of the representations of 1000 samples in the test set of the AP17-OLR extracted by M-nofinetune, and each color represents a language.