Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improved Contextual Recognition In Automatic Speech Recognition Systems By Semantic Lattice Rescoring

Ankitha Sudarshan, Vinay Samuel, Parth Patwa, Ibtihel Amara, Aman Chadha

TL;DR

This paper addresses the challenge of context-dependent transcription in ASR by introducing semantic lattice rescoring, which leverages a Transformer-based language model to re-evaluate lattice paths generated by a GMM-HMM–DNN acoustic framework within the Kaldi ecosystem. The method computes joint likelihoods by integrating acoustic scores with neural LM probabilities, refines word lattices, and demonstrates notable WER reductions on LibriSpeech, including 14% on test-clean and over 21% on test-other. Key contributions include a Transformer-based lattice re-scoring approach, exploration of four lattice types, and empirical evidence that context-aware rescoring improves transcription accuracy beyond similar architectural baselines. The work has practical implications for robust, context-sensitive ASR in real-world settings, with potential extensions to music domains and further hyperparameter optimization.

Abstract

Automatic Speech Recognition (ASR) has witnessed a profound research interest. Recent breakthroughs have given ASR systems different prospects such as faithfully transcribing spoken language, which is a pivotal advancement in building conversational agents. However, there is still an imminent challenge of accurately discerning context-dependent words and phrases. In this work, we propose a novel approach for enhancing contextual recognition within ASR systems via semantic lattice processing leveraging the power of deep learning models in accurately delivering spot-on transcriptions across a wide variety of vocabularies and speaking styles. Our solution consists of using Hidden Markov Models and Gaussian Mixture Models (HMM-GMM) along with Deep Neural Networks (DNN) models integrating both language and acoustic modeling for better accuracy. We infused our network with the use of a transformer-based model to properly rescore the word lattice achieving remarkable capabilities with a palpable reduction in Word Error Rate (WER). We demonstrate the effectiveness of our proposed framework on the LibriSpeech dataset with empirical analyses.

Improved Contextual Recognition In Automatic Speech Recognition Systems By Semantic Lattice Rescoring

TL;DR

This paper addresses the challenge of context-dependent transcription in ASR by introducing semantic lattice rescoring, which leverages a Transformer-based language model to re-evaluate lattice paths generated by a GMM-HMM–DNN acoustic framework within the Kaldi ecosystem. The method computes joint likelihoods by integrating acoustic scores with neural LM probabilities, refines word lattices, and demonstrates notable WER reductions on LibriSpeech, including 14% on test-clean and over 21% on test-other. Key contributions include a Transformer-based lattice re-scoring approach, exploration of four lattice types, and empirical evidence that context-aware rescoring improves transcription accuracy beyond similar architectural baselines. The work has practical implications for robust, context-sensitive ASR in real-world settings, with potential extensions to music domains and further hyperparameter optimization.

Abstract

Automatic Speech Recognition (ASR) has witnessed a profound research interest. Recent breakthroughs have given ASR systems different prospects such as faithfully transcribing spoken language, which is a pivotal advancement in building conversational agents. However, there is still an imminent challenge of accurately discerning context-dependent words and phrases. In this work, we propose a novel approach for enhancing contextual recognition within ASR systems via semantic lattice processing leveraging the power of deep learning models in accurately delivering spot-on transcriptions across a wide variety of vocabularies and speaking styles. Our solution consists of using Hidden Markov Models and Gaussian Mixture Models (HMM-GMM) along with Deep Neural Networks (DNN) models integrating both language and acoustic modeling for better accuracy. We infused our network with the use of a transformer-based model to properly rescore the word lattice achieving remarkable capabilities with a palpable reduction in Word Error Rate (WER). We demonstrate the effectiveness of our proposed framework on the LibriSpeech dataset with empirical analyses.
Paper Structure (12 sections, 1 equation, 6 figures, 4 tables)

This paper contains 12 sections, 1 equation, 6 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Background
  5. Lattice Re-scoring
  6. Experimental Details
  7. Data Corpus and Preprocessing
  8. Acoustic Model
  9. Language Model: Deep Neural Network (DNN)
  10. Transformer Model For Lattice Re-scoring
  11. Results and Discussion
  12. Conclusion

Figures (6)

  • Figure 1: Global overview of our framework. Our framework includes audio input, DNN acoustic model,language model integration, lattice creation and alignment, transformer re-scoring, and transcript generation.
  • Figure 2: Lattice re-scoring strategy. Alignment is involved, n-gram language model integration, and Transformer-based re-scoring to enhance contextual features in the final transcript.
  • Figure 3: A generic view of the lattice
  • Figure 4: Magnified view (partial) of the lattice
  • Figure 5: Example of pre-rescoring lattice given an input utterance.
  • ...and 1 more figures