End-to-End Speech Recognition: A Survey

TL;DR

This survey analyzes end-to-end ASR as a unifying neural framework that replaces the traditional modular HMM-based pipeline with integrated architectures. It categorizes E2E models into explicit-alignment (CTC, RNN-T, RNA), implicit-alignment (AED), and hybrid approaches, detailing training, decoding, and LM integration strategies. It highlights architecture advances (Transformers, Conformers, improved encoders/decoders), data augmentation (SpecAugment), and self-supervised pretraining as key drivers of performance, while addressing practical deployment and latency through streaming-endpointing and multi-pass decoding. The paper also discusses the continued relevance of external LMs, internal-LM compensation, and domain adaptation, underscoring both the promise and the remaining research gaps for robust, scalable E2E ASR in diverse conditions.

Abstract

In the last decade of automatic speech recognition (ASR) research, the introduction of deep learning brought considerable reductions in word error rate of more than 50% relative, compared to modeling without deep learning. In the wake of this transition, a number of all-neural ASR architectures were introduced. These so-called end-to-end (E2E) models provide highly integrated, completely neural ASR models, which rely strongly on general machine learning knowledge, learn more consistently from data, while depending less on ASR domain-specific experience. The success and enthusiastic adoption of deep learning accompanied by more generic model architectures lead to E2E models now becoming the prominent ASR approach. The goal of this survey is to provide a taxonomy of E2E ASR models and corresponding improvements, and to discuss their properties and their relation to the classical hidden Markov model (HMM) based ASR architecture. All relevant aspects of E2E ASR are covered in this work: modeling, training, decoding, and external language model integration, accompanied by discussions of performance and deployment opportunities, as well as an outlook into potential future developments.

Figures (10)

• Figure 1: Example alignment sequence for a CTC model with the target sequence $C = (\texttt{s}, \texttt{e}, \texttt{e})$ (right), alongside a (non-deterministic) finite state automaton (FSA) mohri2002weighted (left) representing the set of all valid alignment paths.
• Figure 2: A representation of the CTC model consisting of an encoder which maps the input speech into a higher-level representation, and a softmax layer which predicts frame-level probabilities over the set of output labels and blank.
• Figure 3: An RNN-T Model graves2012sequencegraves2013speech consists of an encoder which transforms the input speech frames into a high-level representation, and a prediction-network which models the sequence of non-blank labels that have been output previously. The prediction network output, $p_{i_t}$, represents the output after producing the previous non-blank label sequence $c_1, \ldots, c_{i_t}$. The joint network produces a probability distribution over the output symbols (augmented with blank) given the prediction network state and a specific encoded frame.
• Figure 4: Example alignment sequence (right) for an RNN-T model with the target sequence $C = (\texttt{s}, \texttt{e}, \texttt{e})$. Horizontal transitions in the image correspond to blank outputs. The FSA (left) represents the set of all valid RNN-T alignment paths.
• Figure 5: An RNA Model sak2017recurrent resembles the RNN-T model graves2012sequencegraves2013speech in terms of the model structure. However, this model is only permitted to output a single label -- either blank, or non-blank -- in a single frame. Unlike RNN-T, the prediction network state in the RNA model, $\mathbf{q}_{t-1}$, depends on the entire alignment sequence $a_{t-1}, \ldots, a_1$. The joint network produces a probability distribution over the output symbols (augmented with blank) given the prediction network state and a specific encoded frame.