Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing CTC-based speech recognition with diverse modeling units

Shiyi Han, Zhihong Lei, Mingbin Xu, Xingyu Na, Zhen Huang

TL;DR

The paper tackles improving CTC-based end-to-end ASR beyond multi-pass rescoring by jointly training with diverse modeling units (phonemes, characters, and logographic representations). It introduces a training-time approach that attaches auxiliary CTC losses to intermediate encoder layers, aligning the unit representations with the encoder depths that best capture their information. Across LibriSpeech and AISHELL-2, the method yields significant WER/CER improvements without added inference cost or a second-pass acoustic model, and uncovers a consistent pattern where phonetic units peak in mid-layers while linguistic units benefit from deeper layers. The findings offer a practical path to more accurate and efficient ASR by integrating heterogeneous modeling units directly into the training objective, with demonstrated effectiveness on both alphabetic and logographic languages.

Abstract

In recent years, the evolution of end-to-end (E2E) automatic speech recognition (ASR) models has been remarkable, largely due to advances in deep learning architectures like transformer. On top of E2E systems, researchers have achieved substantial accuracy improvement by rescoring E2E model's N-best hypotheses with a phoneme-based model. This raises an interesting question about where the improvements come from other than the system combination effect. We examine the underlying mechanisms driving these gains and propose an efficient joint training approach, where E2E models are trained jointly with diverse modeling units. This methodology does not only align the strengths of both phoneme and grapheme-based models but also reveals that using these diverse modeling units in a synergistic way can significantly enhance model accuracy. Our findings offer new insights into the optimal integration of heterogeneous modeling units in the development of more robust and accurate ASR systems.

Enhancing CTC-based speech recognition with diverse modeling units

TL;DR

The paper tackles improving CTC-based end-to-end ASR beyond multi-pass rescoring by jointly training with diverse modeling units (phonemes, characters, and logographic representations). It introduces a training-time approach that attaches auxiliary CTC losses to intermediate encoder layers, aligning the unit representations with the encoder depths that best capture their information. Across LibriSpeech and AISHELL-2, the method yields significant WER/CER improvements without added inference cost or a second-pass acoustic model, and uncovers a consistent pattern where phonetic units peak in mid-layers while linguistic units benefit from deeper layers. The findings offer a practical path to more accurate and efficient ASR by integrating heterogeneous modeling units directly into the training objective, with demonstrated effectiveness on both alphabetic and logographic languages.

Abstract

In recent years, the evolution of end-to-end (E2E) automatic speech recognition (ASR) models has been remarkable, largely due to advances in deep learning architectures like transformer. On top of E2E systems, researchers have achieved substantial accuracy improvement by rescoring E2E model's N-best hypotheses with a phoneme-based model. This raises an interesting question about where the improvements come from other than the system combination effect. We examine the underlying mechanisms driving these gains and propose an efficient joint training approach, where E2E models are trained jointly with diverse modeling units. This methodology does not only align the strengths of both phoneme and grapheme-based models but also reveals that using these diverse modeling units in a synergistic way can significantly enhance model accuracy. Our findings offer new insights into the optimal integration of heterogeneous modeling units in the development of more robust and accurate ASR systems.
Paper Structure (11 sections, 3 equations, 3 figures, 2 tables)

This paper contains 11 sections, 3 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Backbone Model
  4. Acoustic Model Fusion
  5. Multiple modeling units joint training
  6. Experiments
  7. Overview
  8. LibriSpeech
  9. AISHELL-2
  10. Highlights
  11. Conclusion

Figures (3)

  • Figure 1: An illustration of the proposed model with 4 encoder layers and 2 decoder layers. A phoneme $CTC_{ph}$ is added on top of the 2nd encoder layer.
  • Figure 2: WER on LibriSpeech when different $CTC_{unit}$s are attached.
  • Figure 3: CER trend on test-ios partition of AISHELL2