ELP-Adapters: Parameter Efficient Adapter Tuning for Various Speech Processing Tasks

TL;DR

This work tackles the high parameter cost of fine-tuning self-supervised speech models for diverse downstream tasks. It introduces ELP-adapter tuning, which integrates three adapters—E-adapters for fine-grained linguistic features, L-adapters for leveraging lower-layer non-linguistic information, and P-adapters for injecting pseudo features—into a frozen backbone like WavLM. Across ASR, ASV, SER, and SIC, ELP-adapter tuning matches or surpasses full fine-tuning while reducing learnable parameters by roughly an order of magnitude, with notable gains from combining all three adapters. The approach demonstrates strong task transferability, improved efficiency, and insights into layer-wise contributions, offering a practical path to scalable deployment of self-supervised speech models. Limitations include scenarios with abundant data where full fine-tuning may still be favorable, and future work points to automatic pruning, neural architecture search, and multi-modal extensions.

Abstract

Self-supervised learning has emerged as a key approach for learning generic representations from speech data. Despite promising results in downstream tasks such as speech recognition, speaker verification, and emotion recognition, a significant number of parameters is required, which makes fine-tuning for each task memory-inefficient. To address this limitation, we introduce ELP-adapter tuning, a novel method for parameter-efficient fine-tuning using three types of adapter, namely encoder adapters (E-adapters), layer adapters (L-adapters), and a prompt adapter (P-adapter). The E-adapters are integrated into transformer-based encoder layers and help to learn fine-grained speech representations that are effective for speech recognition. The L-adapters create paths from each encoder layer to the downstream head and help to extract non-linguistic features from lower encoder layers that are effective for speaker verification and emotion recognition. The P-adapter appends pseudo features to CNN features to further improve effectiveness and efficiency. With these adapters, models can be quickly adapted to various speech processing tasks. Our evaluation across four downstream tasks using five backbone models demonstrated the effectiveness of the proposed method. With the WavLM backbone, its performance was comparable to or better than that of full fine-tuning on all tasks while requiring 90% fewer learnable parameters.

Figures (9)

• Figure 1: Self-supervised model and conventional fine-tuning methods. (a) Architecture of self-supervised model, which consists of a CNN encoder and $L$ transformer-based encoders. Vanilla transformer encoder, which consists of a multi-head self-attention (MHSA) module and a feedforward network (FFN) with LayerNorm and skip connections, is illustrated. (b) Weight tuning applied to self-supervised model. It freezes all encoders and learns weights $w_{l}$ for each layer. (c) LoRA tuning applied to self-attention module. It freezes weight matrices $W_{q}, W_{k}, W_{v}$ and injects learnable low-rank matrices $A_{q}, B_{q}, A_{k}, B_{k}, A_{v}, B_{v}$. (d) Prefix tuning, which prepends learnable matrices $P_{k}$ and $P_{v}$ to the key and value matrices. (e) Efficient adapter tuning applied to transformer-based encoder. It inserts two learnable adapters $g^{(l)}_{1}$ and $g^{(l)}_{2}$ to each layer, each of which involves two fully connected (FC) layers $f_{\text{fc1}}^{(l)}$ and $f_{\text{fc2}}^{(l)}$.
• Figure 2: Overview of ELP-Adapter tuning. Three types of adapters integrated into the self-supervised model. (a) E-adapters (red) are inserted into each encoder layer to facilitate learning of fine-grained features for ASR. (b) L-adapters (green) create paths from each encoder layer to the downstream head to extract non-linguistic features that are effective for ASV and SER. (c) P-adapter (yellow) injects pseudo features into the output of the CNN encoder to further improve training effectiveness and efficiency. (d) Minimal downstream heads (gray) are designed for each task to apply task-specific loss function.
• Figure 3: Trade-off between number of learnable parameters and ASR performance in terms of WER with number of frozen layers varying from 1 to 12. The WavLM model was used as a backbone model.
• Figure 4: Trade-off between number of learnable parameters and ASV performance in terms of EER. The WavLM model is used as a backbone model.
• Figure 5: Trade-off between number of learnable parameters and SER performance in terms of ER. The WavLM model is used as a backbone model.