Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Task Arithmetic with Support Languages for Low-Resource ASR

Emma Rafkin, Dan DeGenaro, Xiulin Yang

TL;DR

This paper tackles low-resource ASR by leveraging genetically related high-resource languages through task arithmetic on Whisper-based systems, using task vectors and LoRA adapters to merge target-language models with related support-language models. It defines a formal merging operation $tau_S = theta_S - theta$ and $theta_final = theta_T + lambda tau_S$ (and analogous LoRA merging) and optimizes the scaling parameter $lambda$ on target validation data to improve WER. Evaluating on 21 target languages and five unseen ones in the Mozilla CV spontaneous-speech task, the approach yields consistent WER reductions, with smaller Whisper-tiny models often outperforming larger variants and higher gains when support languages are closely related. The study highlights the importance of language similarity, proxy languages, and model selection for effective cross-language transfer in low-resource ASR, pointing to more efficient and data-conscious ASR for underrepresented languages.

Abstract

The development of resource-constrained approaches to automatic speech recognition (ASR) is of great interest due to its broad applicability to many low-resource languages for which there is scant usable data. Existing approaches to many low-resource natural language processing tasks leverage additional data from higher-resource languages that are closely related to a target low-resource language. One increasingly popular approach uses task arithmetic to combine models trained on different tasks to create a model for a task where there is little to no training data. In this paper, we consider training on a particular language to be a task, and we generate task vectors by fine-tuning variants of the Whisper ASR system. For pairings of high- and low-resource languages, we merge task vectors via a linear combination, optimizing the weights of the linear combination on the downstream word error rate on the low-resource target language's validation set. We find that this approach consistently improves performance on the target languages.

Task Arithmetic with Support Languages for Low-Resource ASR

TL;DR

This paper tackles low-resource ASR by leveraging genetically related high-resource languages through task arithmetic on Whisper-based systems, using task vectors and LoRA adapters to merge target-language models with related support-language models. It defines a formal merging operation and (and analogous LoRA merging) and optimizes the scaling parameter on target validation data to improve WER. Evaluating on 21 target languages and five unseen ones in the Mozilla CV spontaneous-speech task, the approach yields consistent WER reductions, with smaller Whisper-tiny models often outperforming larger variants and higher gains when support languages are closely related. The study highlights the importance of language similarity, proxy languages, and model selection for effective cross-language transfer in low-resource ASR, pointing to more efficient and data-conscious ASR for underrepresented languages.

Abstract

The development of resource-constrained approaches to automatic speech recognition (ASR) is of great interest due to its broad applicability to many low-resource languages for which there is scant usable data. Existing approaches to many low-resource natural language processing tasks leverage additional data from higher-resource languages that are closely related to a target low-resource language. One increasingly popular approach uses task arithmetic to combine models trained on different tasks to create a model for a task where there is little to no training data. In this paper, we consider training on a particular language to be a task, and we generate task vectors by fine-tuning variants of the Whisper ASR system. For pairings of high- and low-resource languages, we merge task vectors via a linear combination, optimizing the weights of the linear combination on the downstream word error rate on the low-resource target language's validation set. We find that this approach consistently improves performance on the target languages.
Paper Structure (13 sections, 2 equations, 3 figures, 4 tables)

This paper contains 13 sections, 2 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Automatic Speech Recognition Models
  4. Low-Resource ASR
  5. Methods
  6. Models and Training
  7. Leveraging Related Languages
  8. Data Sources and Munging
  9. Results
  10. Discussion
  11. Conclusion
  12. Hyperparameter Settings
  13. Languages

Figures (3)

  • Figure 1: Incorporating the support language task vector into the whisper-tiny target language model improved all of the applicable target models.
  • Figure 2: Incorporating the support language into the whisper-large via linear combinations of LoRA layers led to WER improvements, though overall performance is worse than whisper-tiny.
  • Figure 3: There is a moderate correlation between the cosine similarity of each target language and its support language and the impact of the support language model (the $\Delta$WER) on the target model.