Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Adapter Tuning of Pre-trained Speech Models for Automatic Speaker Verification

Mufan Sang, John H. L. Hansen

TL;DR

Experimental results on the VoxCeleb1 dataset demonstrate that the proposed adapters surpass fine-tuning and other parameter-efficient transfer learning methods, achieving superior performance while updating only 5% of the parameters.

Abstract

With excellent generalization ability, self-supervised speech models have shown impressive performance on various downstream speech tasks in the pre-training and fine-tuning paradigm. However, as the growing size of pre-trained models, fine-tuning becomes practically unfeasible due to heavy computation and storage overhead, as well as the risk of overfitting. Adapters are lightweight modules inserted into pre-trained models to facilitate parameter-efficient adaptation. In this paper, we propose an effective adapter framework designed for adapting self-supervised speech models to the speaker verification task. With a parallel adapter design, our proposed framework inserts two types of adapters into the pre-trained model, allowing the adaptation of latent features within intermediate Transformer layers and output embeddings from all Transformer layers. We conduct comprehensive experiments to validate the efficiency and effectiveness of the proposed framework. Experimental results on the VoxCeleb1 dataset demonstrate that the proposed adapters surpass fine-tuning and other parameter-efficient transfer learning methods, achieving superior performance while updating only 5% of the parameters.

Efficient Adapter Tuning of Pre-trained Speech Models for Automatic Speaker Verification

TL;DR

Experimental results on the VoxCeleb1 dataset demonstrate that the proposed adapters surpass fine-tuning and other parameter-efficient transfer learning methods, achieving superior performance while updating only 5% of the parameters.

Abstract

With excellent generalization ability, self-supervised speech models have shown impressive performance on various downstream speech tasks in the pre-training and fine-tuning paradigm. However, as the growing size of pre-trained models, fine-tuning becomes practically unfeasible due to heavy computation and storage overhead, as well as the risk of overfitting. Adapters are lightweight modules inserted into pre-trained models to facilitate parameter-efficient adaptation. In this paper, we propose an effective adapter framework designed for adapting self-supervised speech models to the speaker verification task. With a parallel adapter design, our proposed framework inserts two types of adapters into the pre-trained model, allowing the adaptation of latent features within intermediate Transformer layers and output embeddings from all Transformer layers. We conduct comprehensive experiments to validate the efficiency and effectiveness of the proposed framework. Experimental results on the VoxCeleb1 dataset demonstrate that the proposed adapters surpass fine-tuning and other parameter-efficient transfer learning methods, achieving superior performance while updating only 5% of the parameters.
Paper Structure (14 sections, 4 equations, 2 figures, 3 tables)

This paper contains 14 sections, 4 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Self-supervised Pre-trained Speech Models
  4. Adapter-based Tuning
  5. Method
  6. Inner-layer Adapter and Inter-layer Adapter
  7. Parallel Adapter Design
  8. Experiments
  9. Datasets
  10. Implementation Details
  11. Comparison among Transfer Learning Methods
  12. Ablation Study
  13. Evaluation in More Challenging Scenarios
  14. Conclusions

Figures (2)

  • Figure 1: Overview of the pre-trained model and the proposed adapter framework. During fine-tuning, the pre-trained model is frozen, only the Inner-layer Adapter, Inter-layer Adapter, and the SV backend are updated.
  • Figure 2: Detailed architectures of (a) Sequential Adapter and (b) Parallel Adapter.