Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EESEN: End-to-End Speech Recognition using Deep RNN Models and WFST-based Decoding

Yajie Miao, Mohammad Gowayyed, Florian Metze

TL;DR

Eesen presents an end-to-end ASR framework that uses CTC-trained deep bidirectional RNNs coupled with WFST-based decoding to seamlessly integrate lexicons and language models. By eliminating frame-level labeling and employing a WFST search graph, it achieves competitive WER with traditional HMM/DNN systems and significantly faster decoding plus smaller decoding graphs. The work demonstrates strong results on WSJ for both phoneme- and character-based setups and provides an open-source platform for benchmarking end-to-end ASR models. This approach reduces development complexity while enabling flexible incorporation of linguistic constraints, potentially accelerating deployment across languages and domains.

Abstract

The performance of automatic speech recognition (ASR) has improved tremendously due to the application of deep neural networks (DNNs). Despite this progress, building a new ASR system remains a challenging task, requiring various resources, multiple training stages and significant expertise. This paper presents our Eesen framework which drastically simplifies the existing pipeline to build state-of-the-art ASR systems. Acoustic modeling in Eesen involves learning a single recurrent neural network (RNN) predicting context-independent targets (phonemes or characters). To remove the need for pre-generated frame labels, we adopt the connectionist temporal classification (CTC) objective function to infer the alignments between speech and label sequences. A distinctive feature of Eesen is a generalized decoding approach based on weighted finite-state transducers (WFSTs), which enables the efficient incorporation of lexicons and language models into CTC decoding. Experiments show that compared with the standard hybrid DNN systems, Eesen achieves comparable word error rates (WERs), while at the same time speeding up decoding significantly.

EESEN: End-to-End Speech Recognition using Deep RNN Models and WFST-based Decoding

TL;DR

Eesen presents an end-to-end ASR framework that uses CTC-trained deep bidirectional RNNs coupled with WFST-based decoding to seamlessly integrate lexicons and language models. By eliminating frame-level labeling and employing a WFST search graph, it achieves competitive WER with traditional HMM/DNN systems and significantly faster decoding plus smaller decoding graphs. The work demonstrates strong results on WSJ for both phoneme- and character-based setups and provides an open-source platform for benchmarking end-to-end ASR models. This approach reduces development complexity while enabling flexible incorporation of linguistic constraints, potentially accelerating deployment across languages and domains.

Abstract

The performance of automatic speech recognition (ASR) has improved tremendously due to the application of deep neural networks (DNNs). Despite this progress, building a new ASR system remains a challenging task, requiring various resources, multiple training stages and significant expertise. This paper presents our Eesen framework which drastically simplifies the existing pipeline to build state-of-the-art ASR systems. Acoustic modeling in Eesen involves learning a single recurrent neural network (RNN) predicting context-independent targets (phonemes or characters). To remove the need for pre-generated frame labels, we adopt the connectionist temporal classification (CTC) objective function to infer the alignments between speech and label sequences. A distinctive feature of Eesen is a generalized decoding approach based on weighted finite-state transducers (WFSTs), which enables the efficient incorporation of lexicons and language models into CTC decoding. Experiments show that compared with the standard hybrid DNN systems, Eesen achieves comparable word error rates (WERs), while at the same time speeding up decoding significantly.

Paper Structure

This paper contains 14 sections, 8 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. The EESEN Framework: Model Training
  3. Deep Bidirectional Recurrent Neural Networks
  4. Training with Connectionist Temporal Classification
  5. GPU Implementation
  6. The EESEN Framework: Decoding
  7. Decoding with WFSTs
  8. Posterior Normalization
  9. Experiments
  10. Experimental Setup
  11. Phoneme-based Systems
  12. Character-based Systems
  13. Conclusions and Future Work
  14. Acknowledgements

Figures (5)

  • Figure 1: A memory block of LSTM.
  • Figure 2: A toy example of the grammar (language model) WFST. The arc weights are the probability of emitting the next word when given the previous word. The node 0 is the start node, and the double-circled node is the end node.
  • Figure 3: The WFST for the phoneme-lexicon entry "is IH Z". The "$<$eps$>$" symbol means no inputs are consumed or no outputs are emitted.
  • Figure 4: The WFST for the spelling of the word "is". We allow the word to optionally start and end with the space character "$<$space$>$".
  • Figure 5: An example of the token WFST which depicts the phoneme "IH". We allow the occurrences of the blank label "$<$blank$>$" and the repetitions of the non-blank label "IH".