Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Automatic Speech Recognition in the Modern Era: Architectures, Training, and Evaluation

Md. Nayeem, Md Shamse Tabrej, Kabbojit Jit Deb, Shaonti Goswami, Md. Azizul Hakim

TL;DR

This survey analyzes the modern ASR landscape driven by end-to-end neural architectures and large-scale data paradigms. It covers foundational E2E families (CTC, AED, RNN-T) and their evolution to Transformer/Conformer models, alongside training shifts from fully supervised to self-supervised and weakly supervised learning (e.g., wav2vec 2.0, Whisper). It also compiles datasets, benchmarks, evaluation metrics, and deployment considerations for streaming and on-device use, while addressing robustness, fairness, and privacy. The work synthesizes architectural and data-driven progress to illuminate practical implications and future directions, including multilinguality, personalized privacy-preserving systems, and beyond-WER evaluation metrics.

Abstract

Automatic Speech Recognition (ASR) has undergone a profound transformation over the past decade, driven by advances in deep learning. This survey provides a comprehensive overview of the modern era of ASR, charting its evolution from traditional hybrid systems, such as Gaussian Mixture Model-Hidden Markov Models (GMM-HMMs) and Deep Neural Network-HMMs (DNN-HMMs), to the now-dominant end-to-end neural architectures. We systematically review the foundational end-to-end paradigms: Connectionist Temporal Classification (CTC), attention-based encoder-decoder models, and the Recurrent Neural Network Transducer (RNN-T), which established the groundwork for fully integrated speech-to-text systems. We then detail the subsequent architectural shift towards Transformer and Conformer models, which leverage self-attention to capture long-range dependencies with high computational efficiency. A central theme of this survey is the parallel revolution in training paradigms. We examine the progression from fully supervised learning, augmented by techniques like SpecAugment, to the rise of self-supervised learning (SSL) with foundation models such as wav2vec 2.0, which drastically reduce the reliance on transcribed data. Furthermore, we analyze the impact of largescale, weakly supervised models like Whisper, which achieve unprecedented robustness through massive data diversity. The paper also covers essential ecosystem components, including key datasets and benchmarks (e.g., LibriSpeech, Switchboard, CHiME), standard evaluation metrics (e.g., Word Error Rate), and critical considerations for real-world deployment, such as streaming inference, on-device efficiency, and the ethical imperatives of fairness and robustness. We conclude by outlining open challenges and future research directions.

Automatic Speech Recognition in the Modern Era: Architectures, Training, and Evaluation

TL;DR

This survey analyzes the modern ASR landscape driven by end-to-end neural architectures and large-scale data paradigms. It covers foundational E2E families (CTC, AED, RNN-T) and their evolution to Transformer/Conformer models, alongside training shifts from fully supervised to self-supervised and weakly supervised learning (e.g., wav2vec 2.0, Whisper). It also compiles datasets, benchmarks, evaluation metrics, and deployment considerations for streaming and on-device use, while addressing robustness, fairness, and privacy. The work synthesizes architectural and data-driven progress to illuminate practical implications and future directions, including multilinguality, personalized privacy-preserving systems, and beyond-WER evaluation metrics.

Abstract

Automatic Speech Recognition (ASR) has undergone a profound transformation over the past decade, driven by advances in deep learning. This survey provides a comprehensive overview of the modern era of ASR, charting its evolution from traditional hybrid systems, such as Gaussian Mixture Model-Hidden Markov Models (GMM-HMMs) and Deep Neural Network-HMMs (DNN-HMMs), to the now-dominant end-to-end neural architectures. We systematically review the foundational end-to-end paradigms: Connectionist Temporal Classification (CTC), attention-based encoder-decoder models, and the Recurrent Neural Network Transducer (RNN-T), which established the groundwork for fully integrated speech-to-text systems. We then detail the subsequent architectural shift towards Transformer and Conformer models, which leverage self-attention to capture long-range dependencies with high computational efficiency. A central theme of this survey is the parallel revolution in training paradigms. We examine the progression from fully supervised learning, augmented by techniques like SpecAugment, to the rise of self-supervised learning (SSL) with foundation models such as wav2vec 2.0, which drastically reduce the reliance on transcribed data. Furthermore, we analyze the impact of largescale, weakly supervised models like Whisper, which achieve unprecedented robustness through massive data diversity. The paper also covers essential ecosystem components, including key datasets and benchmarks (e.g., LibriSpeech, Switchboard, CHiME), standard evaluation metrics (e.g., Word Error Rate), and critical considerations for real-world deployment, such as streaming inference, on-device efficiency, and the ethical imperatives of fairness and robustness. We conclude by outlining open challenges and future research directions.
Paper Structure (37 sections, 2 equations, 4 figures, 3 tables)

This paper contains 37 sections, 2 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Architectures and Decoding
  4. Connectionist Temporal Classification (CTC)
  5. Attention-Based Encoder-Decoder (AED)
  6. Recurrent Neural Network Transducer (RNN-T)
  7. Transformer and Conformer Architectures
  8. Decoding and Language Model Fusion
  9. Open-Source Toolkits
  10. Training Paradigms
  11. Supervised Learning and Data Augmentation
  12. Regularization and Curriculum Learning
  13. Self-Supervised Learning (SSL)
  14. Large-Scale Weak Supervision
  15. Datasets and Benchmarks
  16. ...and 22 more sections

Figures (4)

  • Figure 1: A generalized overview of modern end-to-end ASR pipelines. An acoustic encoder processes audio features, and one of several decoding mechanisms (CTC, attention-based decoder, or RNN-T joint network) generates the output text. An external language model can optionally be used to rescore hypotheses.
  • Figure 2: A conceptual landscape of prominent ASR datasets, plotted by corpus size versus the degree of speech spontaneity. Bubble size indicates multilingual coverage.
  • Figure 3: Comparison of published Word Error Rates (WERs) for leading ASR models on the LibriSpeech 'test-clean' and 'test-other' sets. Lower is better.
  • Figure 4: An illustration of the fundamental trade-off between latency and accuracy in streaming ASR systems. The Pareto frontier represents the set of optimal configurations where one metric cannot be improved without degrading the other.