Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DOTA-ME-CS: Daily Oriented Text Audio-Mandarin English-Code Switching Dataset

Yupei Li, Zifan Wei, Heng Yu, Jiahao Xue, Huichi Zhou, Björn W. Schuller

TL;DR

This work tackles Automatic Speech Recognition for Mandarin-English code-switching by introducing DOTA-ME-CS, an 18.54-hour dataset built from 34 bilingual speakers and enhanced with AI-based timbre, speed, and noise modifications to mirror real-world conditions. The authors provide extensive data analysis on audio and text features, including IPA phoneme frequencies, POS tagging, switching points, and long-recording characteristics, and establish baselines using Whisper, SenseVoice, and Paraformer. Benchmark results reveal robust performance for some models but clear gaps in handling code-switching dynamics, especially under augmented conditions. The dataset aims to accelerate progress in bilingual ASR and is complemented by plans to include spontaneous speech and develop code-switching-optimized models for practical deployment.

Abstract

Code-switching, the alternation between two or more languages within communication, poses great challenges for Automatic Speech Recognition (ASR) systems. Existing models and datasets are limited in their ability to effectively handle these challenges. To address this gap and foster progress in code-switching ASR research, we introduce the DOTA-ME-CS: Daily oriented text audio Mandarin-English code-switching dataset, which consists of 18.54 hours of audio data, including 9,300 recordings from 34 participants. To enhance the dataset's diversity, we apply artificial intelligence (AI) techniques such as AI timbre synthesis, speed variation, and noise addition, thereby increasing the complexity and scalability of the task. The dataset is carefully curated to ensure both diversity and quality, providing a robust resource for researchers addressing the intricacies of bilingual speech recognition with detailed data analysis. We further demonstrate the dataset's potential in future research. The DOTA-ME-CS dataset, along with accompanying code, will be made publicly available.

DOTA-ME-CS: Daily Oriented Text Audio-Mandarin English-Code Switching Dataset

TL;DR

This work tackles Automatic Speech Recognition for Mandarin-English code-switching by introducing DOTA-ME-CS, an 18.54-hour dataset built from 34 bilingual speakers and enhanced with AI-based timbre, speed, and noise modifications to mirror real-world conditions. The authors provide extensive data analysis on audio and text features, including IPA phoneme frequencies, POS tagging, switching points, and long-recording characteristics, and establish baselines using Whisper, SenseVoice, and Paraformer. Benchmark results reveal robust performance for some models but clear gaps in handling code-switching dynamics, especially under augmented conditions. The dataset aims to accelerate progress in bilingual ASR and is complemented by plans to include spontaneous speech and develop code-switching-optimized models for practical deployment.

Abstract

Code-switching, the alternation between two or more languages within communication, poses great challenges for Automatic Speech Recognition (ASR) systems. Existing models and datasets are limited in their ability to effectively handle these challenges. To address this gap and foster progress in code-switching ASR research, we introduce the DOTA-ME-CS: Daily oriented text audio Mandarin-English code-switching dataset, which consists of 18.54 hours of audio data, including 9,300 recordings from 34 participants. To enhance the dataset's diversity, we apply artificial intelligence (AI) techniques such as AI timbre synthesis, speed variation, and noise addition, thereby increasing the complexity and scalability of the task. The dataset is carefully curated to ensure both diversity and quality, providing a robust resource for researchers addressing the intricacies of bilingual speech recognition with detailed data analysis. We further demonstrate the dataset's potential in future research. The DOTA-ME-CS dataset, along with accompanying code, will be made publicly available.
Paper Structure (26 sections, 1 equation, 4 figures, 12 tables)

This paper contains 26 sections, 1 equation, 4 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Datasets
  4. SEAME
  5. ASCEND
  6. ASRU
  7. TALCS
  8. ASR models
  9. Dataset collection
  10. Prompts design
  11. Detailed daily categories
  12. Long recordings
  13. Human recording
  14. AI modification
  15. Data analysis
  16. ...and 11 more sections

Figures (4)

  • Figure 1: A Mandarin-English Code-switching example, spanning 2.86 seconds. The top green section displays the raw waveform, the middle red section represents the Mel spectrogram, and the bottom section illustrates the Mel-frequency cepstral coefficients (MFCCs) features.
  • Figure 2: Data collection process of DOTA-ME-CS. Prompts are designed and input into GPT-4o to generate daily-oriented script text. After manually checking for grammatical issues and ensuring the sentences are in code-switching, we invite bilingual volunteers fluent in the switching languages to record their audio. A small subset of the recordings is modified to introduce noise, AI-generated timbres, and speed variations to add diversity and bias to the dataset. Each text-audio pair is uniquely matched to facilitate training for ASR models.
  • Figure 3: Average pitch of each recording in DOTA-ME-CS
  • Figure 4: Maximum pitch of each recording in DOTA-ME-CS