Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TSPC: A Two-Stage Phoneme-Centric Architecture for code-switching Vietnamese-English Speech Recognition

Tran Nguyen Anh, Truong Dinh Dung, Vo Van Nam, Minh N. H. Nguyen

TL;DR

A novel architecture for Vietnamese-English CS ASR, a Two-Stage Phoneme-Centric model (TSPC), which adopts a phoneme-centric approach based on an extended Vietnamese phoneme set as an intermediate representation for mixed-lingual modeling, while remaining efficient under low computational-resource constraints.

Abstract

Code-switching (CS) presents a significant challenge for general Auto-Speech Recognition (ASR) systems. Existing methods often fail to capture the sub tle phonological shifts inherent in CS scenarios. The challenge is particu larly difficult for language pairs like Vietnamese and English, where both distinct phonological features and the ambiguity arising from similar sound recognition are present. In this paper, we propose a novel architecture for Vietnamese-English CS ASR, a Two-Stage Phoneme-Centric model (TSPC). TSPC adopts a phoneme-centric approach based on an extended Vietnamese phoneme set as an intermediate representation for mixed-lingual modeling, while remaining efficient under low computational-resource constraints. Ex perimental results demonstrate that TSPC consistently outperforms exist ing baselines, including PhoWhisper-base, in Vietnamese-English CS ASR, achieving a significantly lower word error rate of 19.06% with reduced train ing resources. Furthermore, the phonetic-based two-stage architecture en ables phoneme adaptation and language conversion to enhance ASR perfor mance in complex CS Vietnamese-English ASR scenarios.

TSPC: A Two-Stage Phoneme-Centric Architecture for code-switching Vietnamese-English Speech Recognition

TL;DR

A novel architecture for Vietnamese-English CS ASR, a Two-Stage Phoneme-Centric model (TSPC), which adopts a phoneme-centric approach based on an extended Vietnamese phoneme set as an intermediate representation for mixed-lingual modeling, while remaining efficient under low computational-resource constraints.

Abstract

Code-switching (CS) presents a significant challenge for general Auto-Speech Recognition (ASR) systems. Existing methods often fail to capture the sub tle phonological shifts inherent in CS scenarios. The challenge is particu larly difficult for language pairs like Vietnamese and English, where both distinct phonological features and the ambiguity arising from similar sound recognition are present. In this paper, we propose a novel architecture for Vietnamese-English CS ASR, a Two-Stage Phoneme-Centric model (TSPC). TSPC adopts a phoneme-centric approach based on an extended Vietnamese phoneme set as an intermediate representation for mixed-lingual modeling, while remaining efficient under low computational-resource constraints. Ex perimental results demonstrate that TSPC consistently outperforms exist ing baselines, including PhoWhisper-base, in Vietnamese-English CS ASR, achieving a significantly lower word error rate of 19.06% with reduced train ing resources. Furthermore, the phonetic-based two-stage architecture en ables phoneme adaptation and language conversion to enhance ASR perfor mance in complex CS Vietnamese-English ASR scenarios.

Paper Structure

This paper contains 18 sections, 3 equations, 8 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Phoneme-Centric Model
  3. Unified Vietnamese Phoneme Representation
  4. Phoneme Conversion Procedure
  5. Dataset Preparation
  6. Two-stage Model Development
  7. Speech-to-Phone
  8. Phone-to-Text
  9. Two-Stage Phoneme-Centric Model
  10. Training Objective
  11. Experiments
  12. Training Details
  13. Experimental Results
  14. Code-switching speech recognition
  15. Vietnamese speech recognition
  16. ...and 3 more sections

Figures (8)

  • Figure 1: Code-switching example in Vietnamese, where English is transformed into a Vietnamese syllable (red).
  • Figure 2: Overlapping phonological systems of Vietnamese and English
  • Figure 3: t-SNE visualization of English and Vietnamese sound similarities, we use PhoWhisper-base encoder as mono-lingual model for embedding audio feature.
  • Figure 4: Pipeline processing of phoneme.
  • Figure 5: Details of data curation and processing including Phone-to-Text and Speech dataset
  • ...and 3 more figures