PIER: A Novel Metric for Evaluating What Matters in Code-Switching

TL;DR

This work targets the inadequacy of Word-Error-Rate (WER) for evaluating code-switching (CSW) in ASR and introduces Point-of-Interest Error Rate (PIER), a metric that focuses error assessment on embedded-language words via alignment-based analysis. Using multiple competitive models (CTC and Encoder-Decoder frameworks) and diverse CSW datasets, the authors demonstrate that monolingual fine-tuning can improve WER while simultaneously worsening CSW performance as captured by PIER. They formalize PIER as $PIER = \frac{|\mathcal{A}_{\mathcal{I}}|}{|\mathcal{I}|}$ and show that PIER provides a more accurate and interpretable view of CSW capabilities, including intra-word vs inter-word switching. The paper argues for adopting PIER to drive more targeted improvements in CSW ASR and provides open-source tooling to enable standardized evaluation across datasets and language pairs.

Abstract

Code-switching, the alternation of languages within a single discourse, presents a significant challenge for Automatic Speech Recognition. Despite the unique nature of the task, performance is commonly measured with established metrics such as Word-Error-Rate (WER). However, in this paper, we question whether these general metrics accurately assess performance on code-switching. Specifically, using both Connectionist-Temporal-Classification and Encoder-Decoder models, we show fine-tuning on non-code-switched data from both matrix and embedded language improves classical metrics on code-switching test sets, although actual code-switched words worsen (as expected). Therefore, we propose Point-of-Interest Error Rate (PIER), a variant of WER that focuses only on specific words of interest. We instantiate PIER on code-switched utterances and show that this more accurately describes the code-switching performance, showing huge room for improvement in future work. This focused evaluation allows for a more precise assessment of model performance, particularly in challenging aspects such as inter-word and intra-word code-switching.

Figures (2)

• Figure 1: W-large
• Figure 2: MMS