Adapting Language Balance in Code-Switching Speech

TL;DR

Code-switching ASR suffers from under-representation of embedded-language tokens; the authors propose a differentiable, script-aware loss that weights embedded tokens more heavily during training to align optimization with the Point-of-Interest Error Rate (PIER). They implement this by a per-token script map and a lightweight weighted cross-entropy objective that adds no inference overhead. In experiments on ArzEn and Mandarin-English (ASCEND/SEAME), weights such as 1.5 or 1.7 yield substantive PIER reductions while preserving WER/MER, and a breakdown confirms fewer substitutions and deletions for embedded tokens. The approach is simple, generalizes across corpora, and offers a practical path to more robust code-switching ASR.

Abstract

Despite achieving impressive results on standard benchmarks, large foundational models still struggle against code-switching test cases. When data scarcity cannot be used as the usual justification for poor performance, the reason may lie in the infrequent occurrence of code-switched moments, where the embedding of the second language appears subtly. Instead of expecting the models to learn this infrequency on their own, it might be beneficial to provide the training process with labels. Evaluating model performance on code-switching data requires careful localization of code-switching points where recognition errors are most consequential, so that the analysis emphasizes mistakes occurring at those moments. Building on this observation, we leverage the difference between the embedded and the main language to highlight those code-switching points and thereby emphasize learning at those locations. This simple yet effective differentiable surrogate mitigates context bias during generation -- the central challenge in code-switching -- thereby improving the model's robustness. Our experiments with Arabic and Chinese-English showed that the models are able to predict the switching places more correctly, reflected by the reduced substitution error.

Figures (1)

• Figure 1: Breakdown of insertion (I), deletion (D), and substitution (S) error counts on Arzen. We compare the baseline fine-tuned model (FT) with our WL$_{1.5}$, separately for embedded and matrix words.