Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learn and Don't Forget: Adding a New Language to ASR Foundation Models

Mengjie Qian, Siyuan Tang, Rao Ma, Kate M. Knill, Mark J. F. Gales

TL;DR

The paper addresses how to add a new language to ASR foundation models without catastrophic forgetting of existing languages. It compares full fine-tuning, LoRA, Soft Language Code Tuning (SLCT), Soft Prompt Tuning (SPT), and Elastic Weight Consolidation (EWC) using Whisper large-v3 on the FLEURS dataset, focusing on Irish, Asturian, and Sorani Kurdish. EWC is formalized with $L(\theta) = L_B(\theta) + \lambda \sum_i F_i (\theta_i - \theta_{A,i}^*)^2$ and task similarity is analyzed via the Fréchet distance $d^2(\hat F_1, \hat F_2) = \tfrac{1}{2} \| \hat F_1^{1/2} - \hat F_2^{1/2} \|_F$, revealing a trade-off: fine-tuning yields the best new-language performance but substantial forgetting, while EWC and parameter-efficient methods better preserve existing languages with varying costs to the new language. The results provide practical guidance for extending language coverage in foundation ASR models and motivate future work on applying these strategies to speech translation. Overall, selecting the right adaptation strategy depends on the degree of Fisher overlap between tasks and the acceptable balance between new-language gains and forgetting risk.

Abstract

Foundation ASR models often support many languages, e.g. 100 languages in Whisper. However, there has been limited work on integrating an additional, typically low-resource, language, while maintaining performance on the original language set. Fine-tuning, while simple, may degrade the accuracy of the original set. We compare three approaches that exploit adaptation parameters: soft language code tuning, train only the language code; soft prompt tuning, train prepended tokens; and LoRA where a small set of additional parameters are optimised. Elastic Weight Consolidation (EWC) offers an alternative compromise with the potential to maintain performance in specific target languages. Results show that direct fine-tuning yields the best performance for the new language but degrades existing language capabilities. EWC can address this issue for specific languages. If only adaptation parameters are used, the language capabilities are maintained but at the cost of performance in the new language.

Learn and Don't Forget: Adding a New Language to ASR Foundation Models

TL;DR

The paper addresses how to add a new language to ASR foundation models without catastrophic forgetting of existing languages. It compares full fine-tuning, LoRA, Soft Language Code Tuning (SLCT), Soft Prompt Tuning (SPT), and Elastic Weight Consolidation (EWC) using Whisper large-v3 on the FLEURS dataset, focusing on Irish, Asturian, and Sorani Kurdish. EWC is formalized with and task similarity is analyzed via the Fréchet distance , revealing a trade-off: fine-tuning yields the best new-language performance but substantial forgetting, while EWC and parameter-efficient methods better preserve existing languages with varying costs to the new language. The results provide practical guidance for extending language coverage in foundation ASR models and motivate future work on applying these strategies to speech translation. Overall, selecting the right adaptation strategy depends on the degree of Fisher overlap between tasks and the acceptable balance between new-language gains and forgetting risk.

Abstract

Foundation ASR models often support many languages, e.g. 100 languages in Whisper. However, there has been limited work on integrating an additional, typically low-resource, language, while maintaining performance on the original language set. Fine-tuning, while simple, may degrade the accuracy of the original set. We compare three approaches that exploit adaptation parameters: soft language code tuning, train only the language code; soft prompt tuning, train prepended tokens; and LoRA where a small set of additional parameters are optimised. Elastic Weight Consolidation (EWC) offers an alternative compromise with the potential to maintain performance in specific target languages. Results show that direct fine-tuning yields the best performance for the new language but degrades existing language capabilities. EWC can address this issue for specific languages. If only adaptation parameters are used, the language capabilities are maintained but at the cost of performance in the new language.
Paper Structure (14 sections, 2 equations, 2 figures, 5 tables)

This paper contains 14 sections, 2 equations, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Fine-tuning
  4. Efficient Fine-tuning with LoRA
  5. Soft Language Code Tuning
  6. Soft Prompt Tuning
  7. Forgetting and Elastic Weight Consolidation
  8. Experiments
  9. Datasets
  10. Zero-shot ASR and Speech Translation
  11. Language code selection
  12. Efficient fine-tuning
  13. Forgetting and EWC
  14. Conclusion

Figures (2)

  • Figure 1: Illustration of the decoding process for existing and new languages using the default decoding and three methods: (a) reusing an existing language code, (b) using a trained soft language code, and (c) employing prepended soft prompts to capture new domain knowledge. Reference text for Irish audio: "tá cuma chiallmhar air seo".
  • Figure 2: Fisher overlap between Fisher parameters extracted for different languages or datasets.