Sparsely Shared LoRA on Whisper for Child Speech Recognition

TL;DR

The paper tackles adapting Whisper to low-resource child speech using parameter-efficient fine-tuning, examining LoRA and AdaLoRA before introducing Sparsely Shared LoRA (S2-LoRA). S2-LoRA shares low-rank factors across Whisper modules and imposes sparse per-weight rank coefficients, enabling effective adaptation with as little as 0.02% trainable parameters. Empirical results on Chinese child speech show S2-LoRA achieves in-domain performance comparable to AdaLoRA and improves cross-domain generalization, with rank patterns mirroring AdaLoRA. The work highlights efficient, scalable adaptation for ASR in low-resource domains and suggests broad applicability to similar low-resource tasks. The approach leverages learned sparse rank allocation to balance performance and parameter efficiency, offering practical benefits for deployment in resource-constrained settings.

Abstract

Whisper is a powerful automatic speech recognition (ASR) model. Nevertheless, its zero-shot performance on low-resource speech requires further improvement. Child speech, as a representative type of low-resource speech, is leveraged for adaptation. Recently, parameter-efficient fine-tuning (PEFT) in NLP was shown to be comparable and even better than full fine-tuning, while only needing to tune a small set of trainable parameters. However, current PEFT methods have not been well examined for their effectiveness on Whisper. In this paper, only parameter composition types of PEFT approaches such as LoRA and Bitfit are investigated as they do not bring extra inference costs. Different popular PEFT methods are examined. Particularly, we compare LoRA and AdaLoRA and figure out the learnable rank coefficient is a good design. Inspired by the sparse rank distribution allocated by AdaLoRA, a novel PEFT approach Sparsely Shared LoRA (S2-LoRA) is proposed. The two low-rank decomposed matrices are globally shared. Each weight matrix only has to maintain its specific rank coefficients that are constrained to be sparse. Experiments on low-resource Chinese child speech show that with much fewer trainable parameters, S2-LoRA can achieve comparable in-domain adaptation performance to AdaLoRA and exhibit better generalization ability on out-of-domain data. In addition, the rank distribution automatically learned by S2-LoRA is found to have similar patterns to AdaLoRA's allocation.

Figures (3)

• Figure 1: The overall illustration of S2-LoRA. Numbers in the green matrices denote the ranks distributed across all weight matrices of the Whisper encoder. The maximum shared rank is 8. The Whisper medium's encoder with 24 layers is shown as an example where SAM and FFM denote self-attention and feed-forward modules, the decoder operates similarly while including another cross-attention module (CAM).
• Figure 2: Full-FT vs. PEFT on Whisper with different amount of adaptation data zh-C1. Full fine-tuning on Whisper large (large-Full-FT) is not conducted due to limited GPU memory.
• Figure 3: The rank distribution of S2-LoRA ($r=8$) and AdaLoRA. For a fair comparison, the initial rank of AdaLoRA is set to 8, and the target rank is 4. In S2-LoRA, the rank coefficients less than $1e-4$ are masked out. The row order in SAM/CAM follows $\{\mathbf{W}_q, \mathbf{W}_v, \mathbf{W}_k, \mathbf{W}_o\}$, while in FFM follows $\{\mathbf{W}_{fc1}, \mathbf{W}_{fc2}\}$.