Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fast and Accurate Recurrent Neural Network Acoustic Models for Speech Recognition

Haşim Sak, Andrew Senior, Kanishka Rao, Françoise Beaufays

TL;DR

Problem: optimize acoustic models for large vocabulary speech recognition. Approach: assess LSTM RNNs with CTC training, followed by sMBR sequence discriminative training, plus frame stacking, frame-rate reduction, CD phones, and word-level outputs. Contributions: frame stacking and lower frame rate stabilize CTC training and reduce compute; context-dependent phones improve performance; sMBR yields about 10% relative WER reduction; word-level CTC models demonstrate viable recognition without a language model. Impact: yields faster, more accurate LVCSR systems and expands viable modeling units beyond phonemes.

Abstract

We have recently shown that deep Long Short-Term Memory (LSTM) recurrent neural networks (RNNs) outperform feed forward deep neural networks (DNNs) as acoustic models for speech recognition. More recently, we have shown that the performance of sequence trained context dependent (CD) hidden Markov model (HMM) acoustic models using such LSTM RNNs can be equaled by sequence trained phone models initialized with connectionist temporal classification (CTC). In this paper, we present techniques that further improve performance of LSTM RNN acoustic models for large vocabulary speech recognition. We show that frame stacking and reduced frame rate lead to more accurate models and faster decoding. CD phone modeling leads to further improvements. We also present initial results for LSTM RNN models outputting words directly.

Fast and Accurate Recurrent Neural Network Acoustic Models for Speech Recognition

TL;DR

Problem: optimize acoustic models for large vocabulary speech recognition. Approach: assess LSTM RNNs with CTC training, followed by sMBR sequence discriminative training, plus frame stacking, frame-rate reduction, CD phones, and word-level outputs. Contributions: frame stacking and lower frame rate stabilize CTC training and reduce compute; context-dependent phones improve performance; sMBR yields about 10% relative WER reduction; word-level CTC models demonstrate viable recognition without a language model. Impact: yields faster, more accurate LVCSR systems and expands viable modeling units beyond phonemes.

Abstract

We have recently shown that deep Long Short-Term Memory (LSTM) recurrent neural networks (RNNs) outperform feed forward deep neural networks (DNNs) as acoustic models for speech recognition. More recently, we have shown that the performance of sequence trained context dependent (CD) hidden Markov model (HMM) acoustic models using such LSTM RNNs can be equaled by sequence trained phone models initialized with connectionist temporal classification (CTC). In this paper, we present techniques that further improve performance of LSTM RNN acoustic models for large vocabulary speech recognition. We show that frame stacking and reduced frame rate lead to more accurate models and faster decoding. CD phone modeling leads to further improvements. We also present initial results for LSTM RNN models outputting words directly.

Paper Structure

This paper contains 11 sections, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. RNN Acoustic Modeling Techniques
  3. CTC Training
  4. Sequence Discriminative Training
  5. Acoustic Features
  6. Context-Dependent Phones
  7. Word Acoustic Models
  8. Experiments
  9. Data & Models
  10. Results & Discussion
  11. Conclusions

Figures (5)

  • Figure 1: Layer connections in unidirectional (top) and bidirectional (bottom) 5-layer LSTM RNNs.
  • Figure 2: Stacking and subsampling of frames. Acoustic features are generated every 10ms, but are concatenated and down-sampled for input to the network: 8 frames are stacked for unidirectional (top) and 3 for bidirectional models (bottom).
  • Figure 3: Label posteriors estimated by various LSTM RNN models plotted against fixed DNN frame level alignments shown only for labels in the alignment on a heldout utterance 'museums in Chicago'. < b> refers to the blank label.
  • Figure 4: Label posteriors for a unidirectional LSTM RNN model with conventional alignment -- no blank label.
  • Figure 5: 'To become a dietary nutritionist what classes should I take for a two year program in a community college'