UniPET-SPK: A Unified Framework for Parameter-Efficient Tuning of Pre-trained Speech Models for Robust Speaker Verification

TL;DR

The paper tackles the challenge of adapting large self-supervised speech models for speaker verification without full fine-tuning. It introduces three parameter-efficient tuning methods—Inner+Inter Adapters, Deep Speaker Prompting, and a unified gating framework called UniPET-SPK—to enable dynamic, per-layer mixtures of adapters and prompts. Empirical results on VoxCeleb, CN-Celeb1, and 1st 48-UTD show UniPET-SPK consistently outperforms full fine-tuning and other PET methods while updating only about 5.4% of parameters, demonstrating strong cross-domain robustness including forensic scenarios. The work advances practical deployment of large SSL speech models in speaker verification by reducing computational and storage costs without sacrificing performance."

Abstract

With excellent generalization ability, SSL speech models have shown impressive performance on various downstream tasks in the pre-training and fine-tuning paradigm. However, as the size of pre-trained models grows, fine-tuning becomes practically unfeasible due to expanding computation and storage requirements and the risk of overfitting. This study explores parameter-efficient tuning (PET) methods for adapting large-scale pre-trained SSL speech models to speaker verification task. Correspondingly, we propose three PET methods: (i)an adapter-tuning method, (ii)a prompt-tuning method, and (iii)a unified framework that effectively incorporates adapter-tuning and prompt-tuning with a dynamically learnable gating mechanism. First, we propose the Inner+Inter Adapter framework, which inserts two types of adapters into pre-trained models, allowing for adaptation of latent features within the intermediate Transformer layers and output embeddings from all Transformer layers, through a parallel adapter design. Second, we propose the Deep Speaker Prompting method that concatenates trainable prompt tokens into the input space of pre-trained models to guide adaptation. Lastly, we propose the UniPET-SPK, a unified framework that effectively incorporates these two alternate PET methods into a single framework with a dynamic trainable gating mechanism. The proposed UniPET-SPK learns to find the optimal mixture of PET methods to match different datasets and scenarios. We conduct a comprehensive set of experiments on several datasets to validate the effectiveness of the proposed PET methods. Experimental results on VoxCeleb, CN-Celeb, and 1st 48-UTD forensic datasets demonstrate that the proposed UniPET-SPK consistently outperforms the two PET methods, fine-tuning, and other parameter-efficient tuning methods, achieving superior performance while updating only 5.4% of the parameters.

Figures (5)

• Figure 1: Performance of parameter-efficient tuning approaches on Vox1-O. The area of each circle is proportional to the total number of tunable parameters including SV backend.
• Figure 2: Overview of the parameter-efficient tuning methods. (a) Adapter, which inserts learnable lightweight networks into a pre-trained model; (b) Prompt, which adds learnable continuous vectors to warp the hidden states; and (c) UniPET-SPK, which dynamically incorporates Adapter and Prompt with a dynamic gating mechanism.
• Figure 3: Overview of the pre-trained model with Inner+Inter Adapter framework, and the detailed architectures of sequential and parallel adapters. During fine-tuning, the pre-trained model is frozen, only the Inner-layer Adapter, Inter-layer Adapter, and the SV backend are updated.
• Figure 4: Overview of the UniPET-SPK framework. During fine-tuning, the pre-trained model is frozen, only the prepended speaker prompts, Inner-layer Adapter, Inter-layer Adapter, gating layers, and the SV backend are updated.
• Figure 5: Layer weight analysis on the VoxCeleb dataset. Layers from 1 to 12 represent 12 Transformer layers in the WavLM Base+ model. Layer 1 is the one closest to the input.