Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Retrieval Augmented Generation based context discovery for ASR

Dimitrios Siskos, Stavros Papadopoulos, Pablo Peso Parada, Jisi Zhang, Karthikeyan Saravanan, Anastasios Drosou

TL;DR

The paper addresses the challenge of improving ASR accuracy for rare and out-of-vocabulary terms by automatic context discovery. It introduces a model-agnostic retrieval-augmented approach (CB-RAG) that uses embedding-based word retrieval with MiniLM and FAISS, and compares it against two LLM-based strategies: prompt-driven context generation and post-recognition transcript correction (LLM-fix). Across TED-LIUMv3, Earnings21, and SPGISpeech, CB-RAG achieves up to about 17% relative WER reduction over no-context, with near-term feasibility due to low latency, while oracle context can yield up to ~24% reduction. The results show CB-RAG as a scalable, plug-and-play solution that balances accuracy and runtime, making it suitable for real-time or resource-constrained deployments; however, lexical sensitivity and reliance on available candidate contexts remain considerations, whereas LLM-based approaches, though offering higher contextual overlap, are costlier and more sensitive to model prompts and hardware.

Abstract

This work investigates retrieval augmented generation as an efficient strategy for automatic context discovery in context-aware Automatic Speech Recognition (ASR) system, in order to improve transcription accuracy in the presence of rare or out-of-vocabulary terms. However, identifying the right context automatically remains an open challenge. This work proposes an efficient embedding-based retrieval approach for automatic context discovery in ASR. To contextualize its effectiveness, two alternatives based on large language models (LLMs) are also evaluated: (1) large language model (LLM)-based context generation via prompting, and (2) post-recognition transcript correction using LLMs. Experiments on the TED-LIUMv3, Earnings21 and SPGISpeech demonstrate that the proposed approach reduces WER by up to 17% (percentage difference) relative to using no-context, while the oracle context results in a reduction of up to 24.1%.

Retrieval Augmented Generation based context discovery for ASR

TL;DR

The paper addresses the challenge of improving ASR accuracy for rare and out-of-vocabulary terms by automatic context discovery. It introduces a model-agnostic retrieval-augmented approach (CB-RAG) that uses embedding-based word retrieval with MiniLM and FAISS, and compares it against two LLM-based strategies: prompt-driven context generation and post-recognition transcript correction (LLM-fix). Across TED-LIUMv3, Earnings21, and SPGISpeech, CB-RAG achieves up to about 17% relative WER reduction over no-context, with near-term feasibility due to low latency, while oracle context can yield up to ~24% reduction. The results show CB-RAG as a scalable, plug-and-play solution that balances accuracy and runtime, making it suitable for real-time or resource-constrained deployments; however, lexical sensitivity and reliance on available candidate contexts remain considerations, whereas LLM-based approaches, though offering higher contextual overlap, are costlier and more sensitive to model prompts and hardware.

Abstract

This work investigates retrieval augmented generation as an efficient strategy for automatic context discovery in context-aware Automatic Speech Recognition (ASR) system, in order to improve transcription accuracy in the presence of rare or out-of-vocabulary terms. However, identifying the right context automatically remains an open challenge. This work proposes an efficient embedding-based retrieval approach for automatic context discovery in ASR. To contextualize its effectiveness, two alternatives based on large language models (LLMs) are also evaluated: (1) large language model (LLM)-based context generation via prompting, and (2) post-recognition transcript correction using LLMs. Experiments on the TED-LIUMv3, Earnings21 and SPGISpeech demonstrate that the proposed approach reduces WER by up to 17% (percentage difference) relative to using no-context, while the oracle context results in a reduction of up to 24.1%.

Paper Structure

This paper contains 11 sections, 2 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Contextual ASR
  4. Context Construction
  5. Baselines
  6. Experimental Setup
  7. Dataset
  8. ASR Model
  9. Evaluation Metrics
  10. Results
  11. Conclusion

Figures (2)

  • Figure 1: Overview of the proposed context-aware ASR pipeline. Context for each segment is extracted using either an embedding-based retrieval method or LLM-based prompt generation, both conditioned on the preceding $k$ segment captions. Audio is segmented via Voice Activity Detection (VAD). The selected context is provided to a contextual ASR system. Optionally, a post-ASR LLM correction module refines the transcript. The final output is the concatenation of all the individual segment transcripts.
  • Figure 2: Zipf distribution of word frequencies piantadosi2014zipf across all datasets, confirming the characteristic long-tail structure of natural language.