Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Developing a Multilingual Dataset and Evaluation Metrics for Code-Switching: A Focus on Hong Kong's Polylingual Dynamics

Peng Xie, Kani Chen

TL;DR

This work addresses the shortage of code-switching–rich audio data and latency-aware evaluation in multilingual ASR, with a focus on Hong Kong's Cantonese-English dynamics. It introduces MADGF, a multi-agent data-generation framework, to produce the 34.8-hour Mixed Cantonese and English (MCE) dataset spanning 18 topics, and defines FAL, a latency-aware metric that blends Fidelity to Original Audio, transcription accuracy, and latency via the formula $\text{FAL} = \alpha \mathcal{F} + \beta\left(1 - \frac{S_{m} + I_{m} + D_{m}}{N_{m}}\right)\cdot 100 + \gamma \left(1 + \frac{\mathcal{L}-1}{M-1}\cdot(100-1)\right)$. Fine-tuning the Whisper model on MCE yields notable improvements in code-switching and Cantonese recognition, with Small variants achieving strong FAL scores and substantial MER reductions compared to baselines. The work provides a practical resource and evaluation framework aimed at improving real-time, latency-sensitive ASR in polyphonic language environments, impacting both research and real-world multilingual deployments.

Abstract

The existing audio datasets are predominantly tailored towards single languages, overlooking the complex linguistic behaviors of multilingual communities that engage in code-switching. This practice, where individuals frequently mix two or more languages in their daily interactions, is particularly prevalent in multilingual regions such as Hong Kong, China. To bridge this gap, we have developed a 34.8-hour dataset of Mixed Cantonese and English (MCE) audio using our Multi-Agent Data Generation Framework (MADGF). We fine-tuned the open-source multilingual Automatic Speech Recognition (ASR) model, Whisper, with the MCE dataset, leading to impressive zero-shot performance. The traditional metrics overlook important factors such as latency in real-world applications and code-switching scenarios. We have introduced a novel evaluation metric called Fidelity to the Original Audio, Accuracy, and Latency (FAL). This metric aims to overcome the limitations of traditional metrics used to assess ASR systems.

Developing a Multilingual Dataset and Evaluation Metrics for Code-Switching: A Focus on Hong Kong's Polylingual Dynamics

TL;DR

This work addresses the shortage of code-switching–rich audio data and latency-aware evaluation in multilingual ASR, with a focus on Hong Kong's Cantonese-English dynamics. It introduces MADGF, a multi-agent data-generation framework, to produce the 34.8-hour Mixed Cantonese and English (MCE) dataset spanning 18 topics, and defines FAL, a latency-aware metric that blends Fidelity to Original Audio, transcription accuracy, and latency via the formula . Fine-tuning the Whisper model on MCE yields notable improvements in code-switching and Cantonese recognition, with Small variants achieving strong FAL scores and substantial MER reductions compared to baselines. The work provides a practical resource and evaluation framework aimed at improving real-time, latency-sensitive ASR in polyphonic language environments, impacting both research and real-world multilingual deployments.

Abstract

The existing audio datasets are predominantly tailored towards single languages, overlooking the complex linguistic behaviors of multilingual communities that engage in code-switching. This practice, where individuals frequently mix two or more languages in their daily interactions, is particularly prevalent in multilingual regions such as Hong Kong, China. To bridge this gap, we have developed a 34.8-hour dataset of Mixed Cantonese and English (MCE) audio using our Multi-Agent Data Generation Framework (MADGF). We fine-tuned the open-source multilingual Automatic Speech Recognition (ASR) model, Whisper, with the MCE dataset, leading to impressive zero-shot performance. The traditional metrics overlook important factors such as latency in real-world applications and code-switching scenarios. We have introduced a novel evaluation metric called Fidelity to the Original Audio, Accuracy, and Latency (FAL). This metric aims to overcome the limitations of traditional metrics used to assess ASR systems.
Paper Structure (8 sections, 4 equations, 3 figures, 4 tables)

This paper contains 8 sections, 4 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. CORPUS DESIGN
  3. MADGF
  4. FAL
  5. STATISTICS OF MCE DATASET
  6. EXPERIMENTS AND RESULTS
  7. Analysis of Code-switching Misrecognition
  8. Conclusion

Figures (3)

  • Figure 1: Overview of MADGF. MADGF consists of two modules that integrate multiple specialized agents to cooperate in generating text data. The Basic Generation Module utilizes a Creator Agent and an Engineer Agent to generate fundamental data. The Reflection module employs a Reflector Agent to check the grammar and enhance the data.
  • Figure 2: MCE text dataset composition. The MCE dataset covers 18 daily topics and comprises over 16,000 sentences, with each sentence containing a mixture of Traditional Chinese and English characters.
  • Figure 3: MCE audio dataset overivew.