Language Bias in Self-Supervised Learning For Automatic Speech Recognition
Edward Storey, Naomi Harte, Peter Bell
TL;DR
The paper investigates language bias in multilingual SSL ASR (XLS-R) by applying the Lottery Ticket Hypothesis to identify language-specific subnetworks. It demonstrates that, due to imbalanced pretraining data with English dominating, fine-tuning tends to rely on weights from the language with the largest data contribution, often English, across downstream languages. Through pruning experiments at $70\%$, $80\%$, and $90\%$ sparsity and IOU analyses, the authors show that English-derived subnetworks generally yield the lowest $CER$ and that non-English subnetworks contribute less effectively, even for linguistically related tasks. The findings highlight the importance of balancing pretraining data by language and linguistic relationships to mitigate bias and improve truly multilingual SSL ASR performance in open-source models.
Abstract
Self-supervised learning (SSL) is used in deep learning to train on large datasets without the need for expensive labelling of the data. Recently, large Automatic Speech Recognition (ASR) models such as XLS-R have utilised SSL to train on over one hundred different languages simultaneously. However, deeper investigation shows that the bulk of the training data for XLS-R comes from a small number of languages. Biases learned through SSL have been shown to exist in multiple domains, but language bias in multilingual SSL ASR has not been thoroughly examined. In this paper, we utilise the Lottery Ticket Hypothesis (LTH) to identify language-specific subnetworks within XLS-R and test the performance of these subnetworks on a variety of different languages. We are able to show that when fine-tuning, XLS-R bypasses traditional linguistic knowledge and builds only on weights learned from the languages with the largest data contribution to the pretraining data.