Adapting Where It Matters: Depth-Aware Adaptation for Efficient Multilingual Speech Recognition in Low-Resource Languages

TL;DR

This work identifies a U-shaped layer-wise plasticity in multilingual speech foundation models, where early and late decoder layers are more language-specific while middle layers preserve language-agnostic semantics. Leveraging this insight, it introduces DAMA, a depth-aware adaptation framework with three components: (i) a Depth-Aware Rank Schedule that allocates higher adaptation capacity to the early and late layers, (ii) SVD-based Initialization to constrain middle-layer updates, and (iii) Basis-Protected Projection that freezes part of the adaptation to preserve semantic structure. Evaluated on 18 low-resource languages across Common Voice and FLEURS using the Whisper large v2 backbone, DAMA achieves state-of-the-art or matched accuracy with up to 80% fewer trainable parameters and substantial gains in memory and training time, including up to 29% relative WER improvements in extremely low-resource settings. These results demonstrate that structure-aware, layer-specific adaptation can deliver scalable, efficient multilingual ASR without sacrificing performance, especially when data are scarce.

Abstract

Recent speech foundation models excel at multilingual automatic speech recognition (ASR) for high-resource languages, but adapting them to low-resource languages remains challenging due to data scarcity and efficiency constraints. Full-model fine-tuning is computationally expensive and prone to overfitting, while parameter-efficient methods like LoRA apply adaptation uniformly across layers, overlooking internal representations thus compromising effectiveness and efficiency. We analyze multilingual ASR models and reveal a U-shaped adaptability pattern: early and late layers are language-specific and require more adaptation, while intermediate layers retain shared semantics and need less. Building on this observation, we propose DAMA, a Depth-Aware Model Adaptation framework that allocates adaptation capacity according to each layer's role. DAMA also introduces Singular Value Decomposition (SVD)-based initialization to constrain adaptation and preserve the U-shaped pattern, as well as a frozen middle-layer basis for further efficiency. Evaluated on 18 low-resource languages across two benchmark datasets, DAMA matches or surpasses state-of-the-art accuracy with 80% fewer trainable parameters, achieves a 29% error reduction under extreme data scarcity, and significantly improves memory, training time, and computational efficiency over baselines. These results highlight the benefits of structure-aware adaptation for efficient, scalable multilingual ASR.

Figures (4)

• Figure 1: Layer-wise probing for different languages before and after fine-tuning.
• Figure 2: Overview of the DAMA Framework compared with LoRA. (a) The standard LoRA with uniform rank. (b) The DAMA Framework. Specifically, the Depth-Aware Rank schedule allocates high plasticity to the early and late layers, while the Basis-Protected Projection physically locks the middle layers to protect "Semantic Valley". All layers mean all the layers from the decoder. The decoder processes acoustic embeddings from the encoder and transcribes them into output tokens.
• Figure 3: Average Peak GPU memory usage for different adaptation methods across all 10 languages on Common Voice.
• Figure 4: Pareto Frontier analysis of one-epoch adaptation time versus overall average WER.