SSVD-O: Parameter-Efficient Fine-Tuning with Structured SVD for Speech Recognition

TL;DR

SSVD-O tackles the challenge of efficiently adapting large speech foundation models to domain-shift ASR by decoupling input and output space updates through a structured SVD framework. It extends SSVD with a learnable outer transformation, controlled by inner ratio $p$ and outer rank $l$, enabling scalable fine-tuning with few parameters while preserving semantic mappings. Across child speech and regional accents, SSVD-O outperforms LoRA-based approaches and approaches full fine-tuning performance while mitigating forgetting, and the study provides a systematic analysis of parameter budget allocation and learning-forgetting trade-offs. The work demonstrates practical impact for adapting large ASR models to diverse, low-resource domains and informs budget-aware PEFT design for continual learning.

Abstract

Parameter-efficient fine-tuning (PEFT) is a scalable approach for adapting large speech foundation models to new domains. While methods such as LoRA and its state-of-the-art variants reduce adaptation costs, they typically allocate parameters uniformly across model subspaces, which limits their efficiency and scalability in speech applications. Building on our prior work, this paper introduces SSVD-Outer (SSVD-O), an extension of the structured SVD-guided (SSVD) fine-tuning method. SSVD-O combines input acoustic feature space-associated inner transformations with output semantic feature space-associated outer transformations to enable scalable and balanced adaptation. We conduct the first systematic analysis of parameter budget allocation across model subspaces in PEFT for automatic speech recognition (ASR), and investigate the trade-off between learning and forgetting under constrained resources. SSVD-O is benchmarked against LoRA, DoRA, PiSSA, and SSVD on domain-shifted ASR tasks, including child speech and regional accents, across model scales from 0.1B to 2B within the ESPnet framework. Experimental results show that SSVD-O consistently narrows the performance gap to full fine-tuning while improving generalization and mitigating catastrophic forgetting.

Figures (8)

• Figure 1: Word error rate (WER %) versus trainable parameters for PEFT tuning FF layers of OWSM-1B on MyST for 10 epochs.
• Figure 2: WER (%) versus number of trainable parameters for SSVD and SSVD-O with varying combinations of inner transformation ratio $p\in [22\%,100\%]$ and outer transformation rank $l\in[8, 708]$, evaluated on OWSM-1B fine-tuned on the MyST dataset for 10 epochs.
• Figure 3: WER (%) versus number of trainable parameters for SSVD and SSVD-O with varying combinations of inner transformation ratio $p\in [22\%,100\%]$ and outer transformation rank $l\in[8, 1024]$, evaluated on OWLS-2B fine-tuned on the MyST dataset for 5 epochs.
• Figure 4: WER (%) versus number of trainable parameters for SSVD and SSVD-O with varying combinations of inner transformation ratio $p\in [33\%,100\%]$ and outer transformation rank $l\in[8, 256]$, evaluated on OWSM-0.1B fine-tuned on the CGN dataset for 5 epochs.
• Figure 5: Learning vs. Avg. forgetting from Table \ref{['tab:forgetting_0.1B']}, the bottom-left indicates the best trade-off.