Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Large Language Model Should Understand Pinyin for Chinese ASR Error Correction

Yuang Li, Xiaosong Qiao, Xiaofeng Zhao, Huan Zhao, Wei Tang, Min Zhang, Hao Yang

TL;DR

Chinese ASR error correction is challenged by homophony; the paper proposes PY-GEC, a Pinyin-enhanced GEC framework that feeds Pinyin as supplementary input to a large language model and trains with multitask objectives. Using synthetic pseudo-errors and one-best ASR hypotheses, PY-GEC, especially with multitask training, achieves lower CER and higher entity recall on Aishell-1 and Common Voice. Attention and feature-space analyses show Pinyin features receive higher influence and align more closely with text representations, explaining performance gains. The approach uses only text-based synthetic data, enabling scalable training without real ASR data, and points to future work with larger LLMs and multi-modal extensions.

Abstract

Large language models can enhance automatic speech recognition systems through generative error correction. In this paper, we propose Pinyin-enhanced GEC, which leverages Pinyi, the phonetic representation of Mandarin Chinese, as supplementary information to improve Chinese ASR error correction. Our approach only utilizes synthetic errors for training and employs the one-best hypothesis during inference. Additionally, we introduce a multitask training approach involving conversion tasks between Pinyin and text to align their feature spaces. Experiments on the Aishell-1 and the Common Voice datasets demonstrate that our approach consistently outperforms GEC with text-only input. More importantly, we provide intuitive explanations for the effectiveness of PY-GEC and multitask training from two aspects: 1) increased attention weight on Pinyin features; and 2) aligned feature space between Pinyin and text hidden states.

Large Language Model Should Understand Pinyin for Chinese ASR Error Correction

TL;DR

Chinese ASR error correction is challenged by homophony; the paper proposes PY-GEC, a Pinyin-enhanced GEC framework that feeds Pinyin as supplementary input to a large language model and trains with multitask objectives. Using synthetic pseudo-errors and one-best ASR hypotheses, PY-GEC, especially with multitask training, achieves lower CER and higher entity recall on Aishell-1 and Common Voice. Attention and feature-space analyses show Pinyin features receive higher influence and align more closely with text representations, explaining performance gains. The approach uses only text-based synthetic data, enabling scalable training without real ASR data, and points to future work with larger LLMs and multi-modal extensions.

Abstract

Large language models can enhance automatic speech recognition systems through generative error correction. In this paper, we propose Pinyin-enhanced GEC, which leverages Pinyi, the phonetic representation of Mandarin Chinese, as supplementary information to improve Chinese ASR error correction. Our approach only utilizes synthetic errors for training and employs the one-best hypothesis during inference. Additionally, we introduce a multitask training approach involving conversion tasks between Pinyin and text to align their feature spaces. Experiments on the Aishell-1 and the Common Voice datasets demonstrate that our approach consistently outperforms GEC with text-only input. More importantly, we provide intuitive explanations for the effectiveness of PY-GEC and multitask training from two aspects: 1) increased attention weight on Pinyin features; and 2) aligned feature space between Pinyin and text hidden states.
Paper Structure (12 sections, 2 equations, 5 figures, 3 tables)

This paper contains 12 sections, 2 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. PY-GEC and Multitask Training
  4. Pseudo Dataset
  5. Ensemble
  6. Analysis
  7. Experiments
  8. Setups
  9. Results for PY-GEC
  10. Attention Analysis
  11. Feature Space Analysis
  12. Conclusion

Figures (5)

  • Figure 1: The flowchart for PY-GEC.
  • Figure 2: The percentages of good and bad cases. The ASR transcriptions are generated by Whisper-Small on Aishell-1 and Common Voice datasets.
  • Figure 3: The attention scores correspond to each input component.
  • Figure 4: The layer-wise attention scores correspond to each input component. (a) PY-GEC; (b) Multitask + PY-GEC.
  • Figure 5: PCA analysis for the last hidden states that correspond to Text and Pinyin. The hidden states are extracted from the original LLaMA-3-8B-Chinese model and our fine-tuned multitask model.