Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Code-switching Speech Recognition Under the Lens: Model- and Data-Centric Perspectives

Hexin Liu, Haoyang Zhang, Qiquan Zhang, Xiangyu Zhang, Dongyuan Shi, Eng Siong Chng, Haizhou Li

Abstract

Code-switching automatic speech recognition (CS-ASR) presents unique challenges due to language confusion introduced by spontaneous intra-sentence switching and accent bias that blurs the phonetic boundaries. Although the constituent languages may be individually high-resource, the scarcity of annotated code-switching data further compounds these challenges. In this paper, we systematically analyze CS-ASR from both model-centric and data-centric perspectives. By comparing state-of-the-art algorithmic methods, including language-specific processing and auxiliary language-aware multi-task learning, we discuss their varying effectiveness across datasets with different linguistic characteristics. On the data side, we first investigate TTS as a data augmentation method. By varying the textual characteristics and speaker accents, we analyze the impact of language confusion and accent bias on CS-ASR. To further mitigate data scarcity and enhance textual diversity, we propose a prompting strategy by simplifying the equivalence constraint theory (SECT) to guide large language models (LLMs) in generating linguistically valid code-switching text. The proposed SECT outperforms existing methods in ASR performance and linguistic quality assessments, generating code-switching text that more closely resembles real-world code-switching text. When used to generate speech-text pairs via TTS, SECT proves effective in improving CS-ASR performance. Our analysis of both model- and data-centric methods underscores that effective CS-ASR requires strategies to be carefully aligned with the specific linguistic characteristics of the code-switching data.

Code-switching Speech Recognition Under the Lens: Model- and Data-Centric Perspectives

Abstract

Code-switching automatic speech recognition (CS-ASR) presents unique challenges due to language confusion introduced by spontaneous intra-sentence switching and accent bias that blurs the phonetic boundaries. Although the constituent languages may be individually high-resource, the scarcity of annotated code-switching data further compounds these challenges. In this paper, we systematically analyze CS-ASR from both model-centric and data-centric perspectives. By comparing state-of-the-art algorithmic methods, including language-specific processing and auxiliary language-aware multi-task learning, we discuss their varying effectiveness across datasets with different linguistic characteristics. On the data side, we first investigate TTS as a data augmentation method. By varying the textual characteristics and speaker accents, we analyze the impact of language confusion and accent bias on CS-ASR. To further mitigate data scarcity and enhance textual diversity, we propose a prompting strategy by simplifying the equivalence constraint theory (SECT) to guide large language models (LLMs) in generating linguistically valid code-switching text. The proposed SECT outperforms existing methods in ASR performance and linguistic quality assessments, generating code-switching text that more closely resembles real-world code-switching text. When used to generate speech-text pairs via TTS, SECT proves effective in improving CS-ASR performance. Our analysis of both model- and data-centric methods underscores that effective CS-ASR requires strategies to be carefully aligned with the specific linguistic characteristics of the code-switching data.

Paper Structure

This paper contains 24 sections, 5 equations, 4 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Model-centric methods
  3. Unified model and auxiliary tasks
  4. Language-specific Modules
  5. Data-centric analysis
  6. Data Scarcity and Monolingual Utilization
  7. Code-switching speech synthesis
  8. Code-switching text generation
  9. Datasets and experiments
  10. Datasets
  11. Experiment setup and model configuration
  12. Performance measurement
  13. Results and Analysis
  14. Comparison of model-centric methods
  15. Impact of Pre-training
  16. ...and 9 more sections

Figures (4)

  • Figure 1: Overview of existing methods in code-switching ASR.
  • Figure 2: Improved CS-ASR algorithms with (a) multi-task learning with auxiliary task or (b) language-specific modules, where blue and green colors denote two languages, long and short units denote speech and text representations, respectively.
  • Figure 3: The CosyVoice 2 model used for TTS, only the LLM module within CosyVoice 2 is updated during fine-tuning.
  • Figure 4: An example of translating Mandarin text to English-Mandarin Code-switching text via SECT-guided LLM, the proposed SECT-based prompt is shown in the dashed block.