HiKE: Hierarchical Evaluation Framework for Korean-English Code-Switching Speech Recognition

TL;DR

This work addresses the lack of robust evaluation resources for Korean–English code-switching ASR by introducing HiKE, a publicly accessible CS benchmark with 1,121 natural utterances, topic diversity, loanword labeling, and three-level CS annotations (word, phrase, sentence). It combines a human–LLM data creation workflow with meticulous recording and labeling to enable precise assessment of CS handling, using MER and PIER as evaluation metrics. A broad evaluation across 9 multilingual ASR models reveals substantial CS degradation compared to monolingual performance, with scale and architecture influencing CS-level strengths and weaknesses. Fine-tuning experiments show that CS capability can be effectively enhanced through both natural intra-sentential CS data and synthetic inter-sentential CS data created by concatenating monolingual utterances, suggesting cost-effective paths for improving CS‑ASR in resource-constrained settings. HiKE thus provides a foundation for future research on CS–ASR across language pairs, data synthesis methods, and scalable benchmarking, with open-source release to accelerate progress.

Abstract

Despite advances in multilingual automatic speech recognition (ASR), code-switching (CS), the mixing of languages within an utterance common in daily speech, remains a severely underexplored challenge. In this paper, we introduce HiKE: the Hierarchical Korean-English code-switching benchmark, the first globally accessible evaluation framework for Korean-English CS, aiming to provide a means for the precise evaluation of multilingual ASR models and to foster research in the field. The proposed framework not only consists of high-quality, natural CS data across various topics, but also provides meticulous loanword labels and a hierarchical CS-level labeling scheme (word, phrase, and sentence) that together enable a systematic evaluation of a model's ability to handle each distinct level of code-switching. Through evaluations of diverse multilingual ASR models and fine-tuning experiments, this paper demonstrates that although most multilingual ASR models initially exhibit inadequate CS-ASR performance, this capability can be enabled through fine-tuning with synthetic CS data. HiKE is available at https://github.com/ThetaOne-AI/HiKE

Figures (3)

• Figure 1: Code-Switching Examples by CS-Level
• Figure 2: CS-ASR Fail Cases
• Figure 3: Screenshot of Recording Tool