Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An efficient text augmentation approach for contextualized Mandarin speech recognition

Naijun Zheng, Xucheng Wan, Kai Liu, Ziqing Du, Zhou Huan

TL;DR

The paper addresses rare-word recognition in Mandarin ASR under limited paired data and computational budgets. It introduces a text augmentation framework that constructs small codebooks from text data and integrates latent text embeddings into a CIF-based contextualized ASR without fine-tuning. In experiments on Wenetspeech and AIShell, the TA method consistently outperforms baselines, achieving substantial improvements in both general CER and hotword bias metrics, including up to 30% relative gains on rare words and about 15% relative gains in overall CER. The approach is cost-efficient, relies primarily on text data, and generalizes to other E2E backbones, with future work on coding-switched hotwords and broader backbone applicability.

Abstract

Although contextualized automatic speech recognition (ASR) systems are commonly used to improve the recognition of uncommon words, their effectiveness is hindered by the inherent limitations of speech-text data availability. To address this challenge, our study proposes to leverage extensive text-only datasets and contextualize pre-trained ASR models using a straightforward text-augmentation (TA) technique, all while keeping computational costs minimal. In particular, to contextualize a pre-trained CIF-based ASR, we construct a codebook using limited speech-text data. By utilizing a simple codebook lookup process, we convert available text-only data into latent text embeddings. These embeddings then enhance the inputs for the contextualized ASR. Our experiments on diverse Mandarin test sets demonstrate that our TA approach significantly boosts recognition performance. The top-performing system shows relative CER improvements of up to 30% on rare words and 15% across all words in general.

An efficient text augmentation approach for contextualized Mandarin speech recognition

TL;DR

The paper addresses rare-word recognition in Mandarin ASR under limited paired data and computational budgets. It introduces a text augmentation framework that constructs small codebooks from text data and integrates latent text embeddings into a CIF-based contextualized ASR without fine-tuning. In experiments on Wenetspeech and AIShell, the TA method consistently outperforms baselines, achieving substantial improvements in both general CER and hotword bias metrics, including up to 30% relative gains on rare words and about 15% relative gains in overall CER. The approach is cost-efficient, relies primarily on text data, and generalizes to other E2E backbones, with future work on coding-switched hotwords and broader backbone applicability.

Abstract

Although contextualized automatic speech recognition (ASR) systems are commonly used to improve the recognition of uncommon words, their effectiveness is hindered by the inherent limitations of speech-text data availability. To address this challenge, our study proposes to leverage extensive text-only datasets and contextualize pre-trained ASR models using a straightforward text-augmentation (TA) technique, all while keeping computational costs minimal. In particular, to contextualize a pre-trained CIF-based ASR, we construct a codebook using limited speech-text data. By utilizing a simple codebook lookup process, we convert available text-only data into latent text embeddings. These embeddings then enhance the inputs for the contextualized ASR. Our experiments on diverse Mandarin test sets demonstrate that our TA approach significantly boosts recognition performance. The top-performing system shows relative CER improvements of up to 30% on rare words and 15% across all words in general.
Paper Structure (12 sections, 4 equations, 4 figures, 2 tables)

This paper contains 12 sections, 4 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Proposed Method
  4. Framework pipeline
  5. Text sampler module
  6. Optional Variation Adaption (VA) schemes
  7. Training and inference
  8. Experiments
  9. Data sets
  10. Experimental setup
  11. Experimental results
  12. Conclusion

Figures (4)

  • Figure 1: Overall diagram of proposed TA-enhanced contextualized ASR system: (a) CIF-based ASR; (b) Text sampler; (c) Training framework with TA-enhanced biasing module.
  • Figure 2: Distribution plots for text embeddings of Chinese characters, where each point is labeled and colored according to the first syllable in Pinyin. (Best viewed in color)
  • Figure 3: The impact of $\lambda$ on CER.
  • Figure 4: Comparisons of various VA schemes with CIF bias.