Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Transformer-based Acoustic Modeling for Hybrid Speech Recognition

Yongqiang Wang, Abdelrahman Mohamed, Duc Le, Chunxi Liu, Alex Xiao, Jay Mahadeokar, Hongzhao Huang, Andros Tjandra, Xiaohui Zhang, Frank Zhang, Christian Fuegen, Geoffrey Zweig, Michael L. Seltzer

TL;DR

The paper targets improving acoustic modeling for hybrid ASR beyond recurrent architectures. It introduces transformer-based acoustic encoders with different positional embeddings, an iterated loss to train deep transformers, and a study of limited right-context for streaming. On Librispeech, transformer AMs achieve 19-26% relative WER reductions with a 4-gram LM and reach state-of-the-art results with neural LM rescoring, with corroboration on a large internal dataset. These results suggest transformer-based AMs are a strong candidate for practical hybrid ASR and point to remaining challenges in streaming efficiency and scaling.

Abstract

We propose and evaluate transformer-based acoustic models (AMs) for hybrid speech recognition. Several modeling choices are discussed in this work, including various positional embedding methods and an iterated loss to enable training deep transformers. We also present a preliminary study of using limited right context in transformer models, which makes it possible for streaming applications. We demonstrate that on the widely used Librispeech benchmark, our transformer-based AM outperforms the best published hybrid result by 19% to 26% relative when the standard n-gram language model (LM) is used. Combined with neural network LM for rescoring, our proposed approach achieves state-of-the-art results on Librispeech. Our findings are also confirmed on a much larger internal dataset.

Transformer-based Acoustic Modeling for Hybrid Speech Recognition

TL;DR

The paper targets improving acoustic modeling for hybrid ASR beyond recurrent architectures. It introduces transformer-based acoustic encoders with different positional embeddings, an iterated loss to train deep transformers, and a study of limited right-context for streaming. On Librispeech, transformer AMs achieve 19-26% relative WER reductions with a 4-gram LM and reach state-of-the-art results with neural LM rescoring, with corroboration on a large internal dataset. These results suggest transformer-based AMs are a strong candidate for practical hybrid ASR and point to remaining challenges in streaming efficiency and scaling.

Abstract

We propose and evaluate transformer-based acoustic models (AMs) for hybrid speech recognition. Several modeling choices are discussed in this work, including various positional embedding methods and an iterated loss to enable training deep transformers. We also present a preliminary study of using limited right context in transformer models, which makes it possible for streaming applications. We demonstrate that on the widely used Librispeech benchmark, our transformer-based AM outperforms the best published hybrid result by 19% to 26% relative when the standard n-gram language model (LM) is used. Combined with neural network LM for rescoring, our proposed approach achieves state-of-the-art results on Librispeech. Our findings are also confirmed on a much larger internal dataset.

Paper Structure

This paper contains 16 sections, 3 equations, 1 figure, 6 tables.

Table of Contents

  1. Introduction
  2. Hybrid Architecture
  3. Acoustic Modeling Using Transformer
  4. Self-Attention and Multi-Head Attention
  5. Architecture of Transformer
  6. Positional Embedding
  7. Training Deep Transformers
  8. Relation to Other Works
  9. Experiments
  10. Experiment Setups
  11. Effect of Positional Embedding
  12. Transformer vs. BLSTM
  13. Effect of Iterated Loss
  14. Limited Right Context
  15. Large Scale Experiments
  16. ...and 1 more sections

Figures (1)

  • Figure 1: Architecture of one transformer layer. "LN" means layer normalization lei2016layer; "FC" means fully connected linear transformation; "gelu" means the gelu nonlinear activation hendrycks2016gaussian.