Hypernetworks for Personalizing ASR to Atypical Speech

TL;DR

This work tackles ASR personalization for atypical speech, a setting hampered by data scarcity and the need for diagnosis-specific cohorts. It introduces meta-learned hypernetworks that dynamically generate utterance-level adaptation parameters, enabling zero-shot personalization across diverse speech disorders. The approach achieves a $75.2\%$ relative WER reduction with only $0.1\%$ of the full parameter budget and generalizes better to unseen speakers than traditional cohort- or full-fine-tuning. By identifying $W_1$ as the most impactful adaptation parameter and validating across multiple Whisper backbones, the study demonstrates practical, scalable personalization without target-speaker data or explicit cohort labels.

Abstract

Parameter-efficient fine-tuning (PEFT) for personalizing automatic speech recognition (ASR) has recently shown promise for adapting general population models to atypical speech. However, these approaches assume a priori knowledge of the atypical speech disorder being adapted for -- the diagnosis of which requires expert knowledge that is not always available. Even given this knowledge, data scarcity and high inter/intra-speaker variability further limit the effectiveness of traditional fine-tuning. To circumvent these challenges, we first identify the minimal set of model parameters required for ASR adaptation. Our analysis of each individual parameter's effect on adaptation performance allows us to reduce Word Error Rate (WER) by half while adapting 0.03% of all weights. Alleviating the need for cohort-specific models, we next propose the novel use of a meta-learned hypernetwork to generate highly individualized, utterance-level adaptations on-the-fly for a diverse set of atypical speech characteristics. Evaluating adaptation at the global, cohort and individual-level, we show that hypernetworks generalize better to out-of-distribution speakers, while maintaining an overall relative WER reduction of 75.2% using 0.1% of the full parameter budget.

Figures (8)

• Figure 1: Speaker median MER of untuned pre-trained Whisper models on $X_S$ (stuttering), $X_D$ (dysarthria) and $X_P$ (Parkinson's).
• Figure 2: Adaptation magnitudes of key $K$, value $V$, query $Q$, output projection $O$ matrices of the self/cross-attention components, as well as the first $W_1$ and second $W_2$ layers of the MLP within each Whisper (tiny.en) encoder/decoder layer, measured in SSAs according to Section \ref{['sec:ssa']}.
• Figure 3: Speaker median MER on $X_S$ (stuttering), $X_D$ (dysarthria) and $X_P$ (Parkinson's) of Whisper (large-v2) untuned, fully-tuned, and adapted using LoRA at both MLP layers or $W_1$, using $r \in [2;64]$.
• Figure 4: Average speaker WER of Whisper (tiny.en) untuned and best fully/partially/LoRA configurations, across $X_P$ (Parkinson's), $X_S$ (stuttering) and $X_D$ (dysarthria), with various degrees of severity.
• Figure 5: Hypernetwork for adapting $W$ with LoRA weights $A$ and $B$, generated by $\theta_A$ and $\theta_B$, respectively initialized with $\mathcal{N}(0, \sigma^2)$ and zeroes. Generation is conditioned on speaker characteristics $s$ from an audio encoder, and generation context $c$ denoting the target parameter's location. All trainable parameters are within the hypernetwork.