Pitch-Aware RNN-T for Mandarin Chinese Mispronunciation Detection and Diagnosis

TL;DR

Addressing Mandarin MDD with tonal considerations and data scarcity, the paper proposes a stateless RNN-T that fuses HuBERT SSL features with pitch embeddings. It introduces a Pitch Fusion Block and Pitch Encoder to integrate F0-based pitch information, trained on AISHELL-1 and evaluated on LATIC. The training objective is $\mathcal{L}_{RNN-T} = - P (\boldsymbol{y} | \boldsymbol{f}) = - \sum_{\boldsymbol{a} \in \boldsymbol{M^{-1}} (\boldsymbol{y})} P (\boldsymbol{a} | \boldsymbol{f})$. Results show a 3% relative improvement in Phone Error Rate and a 7% increase in False Acceptance Rate over a state-of-the-art baseline in non-native scenarios, with pitch-aware fusion improving precision and F1, and reducing Diagnostic Error Rate for high-recall cases. This work demonstrates the feasibility and impact of end-to-end, pitch-aware MDD for tonal languages and suggests avenues for improving Mandarin L2 feedback and tonal ASR systems.

Abstract

Mispronunciation Detection and Diagnosis (MDD) systems, leveraging Automatic Speech Recognition (ASR), face two main challenges in Mandarin Chinese: 1) The two-stage models create an information gap between the phoneme or tone classification stage and the MDD stage. 2) The scarcity of Mandarin MDD datasets limits model training. In this paper, we introduce a stateless RNN-T model for Mandarin MDD, utilizing HuBERT features with pitch embedding through a Pitch Fusion Block. Our model, trained solely on native speaker data, shows a 3% improvement in Phone Error Rate and a 7% increase in False Acceptance Rate over the state-of-the-art baseline in non-native scenarios

Figures (2)

• Figure 1: The Proposed Tonal Phoneme MDD Framework.
• Figure 2: The architecture of the Pitch Fusion Block. The Multi-Head Self-Attention is designed to capture global pitch features, while the residual convolution blocks (delineated by dotted lines and colored in green) aim to capture local pitch features.