Bayesian Low-Rank Factorization for Robust Model Adaptation

TL;DR

This work tackles the challenge of adapting large multilingual speech foundation models to domain-specific needs like code-switching without eroding their broad capabilities. It introduces Bayesian Low-Rank Adaptation (BLoRA), which imposes a zero-mean, small-variance Gaussian prior on LoRA adapters to produce sparse updates and reduce catastrophic forgetting, implemented on the Whisper model. Empirical results across three code-switching datasets show that BLoRA achieves similar in-domain gains to LoRA while dramatically improving retention of the base model on backward evaluations, and analysis reveals highly sparse, concentrated weight updates under the Bayesian prior. The approach offers a practical, one-step adaptation strategy that preserves generalization and enhances robustness for real-world deployment of speech foundation models.

Abstract

Large speech foundation models achieve strong performance across many domains, but they often require adaptation to handle local needs such as code-switching, where speakers mix languages within the same utterance. Direct fine-tuning of these models risks overfitting to the target domain and overwriting the broad capabilities of the base model. To address this challenge, we explore Bayesian factorized adapters for speech foundation models, which place priors near zero to achieve sparser adaptation matrices and thereby retain general performance while adapting to specific domains. We apply our approach to the Whisper model and evaluate on different multilingual code-switching scenarios. Our results show only minimal adaptation loss while significantly reducing catastrophic forgetting of the base model. Compared to LoRA, our method achieves a backward gain of 54% with only a 4% drop on the new domain. These findings highlight the effectiveness of Bayesian adaptation for fine-tuning speech foundation models without sacrificing generalization.