Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SENS-ASR: Semantic Embedding injection in Neural-transducer for Streaming Automatic Speech Recognition

Youness Dkhissi, Valentin Vielzeuf, Elys Allesiardo, Anthony Larcher

TL;DR

This work presents SENS-ASR, an approach to enhance the transcription quality of Streaming-ASR by reinforcing the acoustic information with semantic information, trained using knowledge distillation from a sentence embedding Language Model fine-tuned on the training dataset transcriptions.

Abstract

Many Automatic Speech Recognition (ASR) applications require streaming processing of the audio data. In streaming mode, ASR systems need to start transcribing the input stream before it is complete, i.e., the systems have to process a stream of inputs with a limited (or no) future context. Compared to offline mode, this reduction of the future context degrades the performance of Streaming-ASR systems, especially while working with low-latency constraint. In this work, we present SENS-ASR, an approach to enhance the transcription quality of Streaming-ASR by reinforcing the acoustic information with semantic information. This semantic information is extracted from the available past frame-embeddings by a context module. This module is trained using knowledge distillation from a sentence embedding Language Model fine-tuned on the training dataset transcriptions. Experiments on standard datasets show that SENS-ASR significantly improves the Word Error Rate on small-chunk streaming scenarios.

SENS-ASR: Semantic Embedding injection in Neural-transducer for Streaming Automatic Speech Recognition

TL;DR

This work presents SENS-ASR, an approach to enhance the transcription quality of Streaming-ASR by reinforcing the acoustic information with semantic information, trained using knowledge distillation from a sentence embedding Language Model fine-tuned on the training dataset transcriptions.

Abstract

Many Automatic Speech Recognition (ASR) applications require streaming processing of the audio data. In streaming mode, ASR systems need to start transcribing the input stream before it is complete, i.e., the systems have to process a stream of inputs with a limited (or no) future context. Compared to offline mode, this reduction of the future context degrades the performance of Streaming-ASR systems, especially while working with low-latency constraint. In this work, we present SENS-ASR, an approach to enhance the transcription quality of Streaming-ASR by reinforcing the acoustic information with semantic information. This semantic information is extracted from the available past frame-embeddings by a context module. This module is trained using knowledge distillation from a sentence embedding Language Model fine-tuned on the training dataset transcriptions. Experiments on standard datasets show that SENS-ASR significantly improves the Word Error Rate on small-chunk streaming scenarios.
Paper Structure (17 sections, 4 equations, 2 figures, 3 tables)

This paper contains 17 sections, 4 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Proposed method
  3. Semantic Context
  4. Teacher Sentence Embedding Model fine-tuning
  5. Paraphrasing protocol
  6. Avoiding neural collapse during fine-tuning
  7. Dynamic Chunk Training
  8. Experiments
  9. Evaluation protocol
  10. Model configuration
  11. Results and discussions
  12. Error analysis
  13. Conclusion
  14. Acknowledgements
  15. Ethical considerations and limitations
  16. ...and 2 more sections

Figures (2)

  • Figure 1: Architecture of the SENS-ASR system using an RNN-T model and a context module. Components in the red dashed-rectangle are only used during training. Components in dashed-circles are the parts of the system global loss.
  • Figure 2: Example of the proposed LLM paraphrasing process. The text in bold refers to the input and the output, while the strikethrough text represents the rejected paraphrases in each pruning step.