Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LA-RAG:Enhancing LLM-based ASR Accuracy with Retrieval-Augmented Generation

Shaojun Li, Hengchao Shang, Daimeng Wei, Jiaxin Guo, Zongyao Li, Xianghui He, Min Zhang, Hao Yang

TL;DR

LA-RAG leverages fine-grained token-level speech datastores and a speech-to-speech retrieval mechanism to enhance ASR accuracy via LLM in-context learning (ICL) capabilities, validating the effectiveness of the approach, especially in handling accent variations.

Abstract

Recent advancements in integrating speech information into large language models (LLMs) have significantly improved automatic speech recognition (ASR) accuracy. However, existing methods often constrained by the capabilities of the speech encoders under varied acoustic conditions, such as accents. To address this, we propose LA-RAG, a novel Retrieval-Augmented Generation (RAG) paradigm for LLM-based ASR. LA-RAG leverages fine-grained token-level speech datastores and a speech-to-speech retrieval mechanism to enhance ASR accuracy via LLM in-context learning (ICL) capabilities. Experiments on Mandarin and various Chinese dialect datasets demonstrate significant improvements in ASR accuracy compared to existing methods, validating the effectiveness of our approach, especially in handling accent variations.

LA-RAG:Enhancing LLM-based ASR Accuracy with Retrieval-Augmented Generation

TL;DR

LA-RAG leverages fine-grained token-level speech datastores and a speech-to-speech retrieval mechanism to enhance ASR accuracy via LLM in-context learning (ICL) capabilities, validating the effectiveness of the approach, especially in handling accent variations.

Abstract

Recent advancements in integrating speech information into large language models (LLMs) have significantly improved automatic speech recognition (ASR) accuracy. However, existing methods often constrained by the capabilities of the speech encoders under varied acoustic conditions, such as accents. To address this, we propose LA-RAG, a novel Retrieval-Augmented Generation (RAG) paradigm for LLM-based ASR. LA-RAG leverages fine-grained token-level speech datastores and a speech-to-speech retrieval mechanism to enhance ASR accuracy via LLM in-context learning (ICL) capabilities. Experiments on Mandarin and various Chinese dialect datasets demonstrate significant improvements in ASR accuracy compared to existing methods, validating the effectiveness of our approach, especially in handling accent variations.
Paper Structure (14 sections, 3 equations, 2 figures, 2 tables)

This paper contains 14 sections, 3 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Speech Tokenizer
  4. Datastore Creation
  5. Speech Retrieval
  6. LLM Prompt
  7. Experimental Setup
  8. Dataset
  9. Implementation Details
  10. Results
  11. Analysis
  12. Retrieval Comparison
  13. Parameter Settings
  14. Conclusion

Figures (2)

  • Figure 1: Overview of proposed LA-RAG, The speech tokenizer is employed to generate aligned speech tokens and text tokens. With the 1th token as an example, the input of A' represents an incorrect token, with the corresponding speech token indicated in green, which is one of retention of N-best pruning. This speech token is subsequently used to query the datastore. The retrieval examples include the mappings between speech token and the correct token A. Ultimately, the examples, the input speech tokens and the N-best results, are transmitted to the LLM prompt for ICL via the adapter and embed process.
  • Figure 2: Left side is the CER trend when use different top k, right side is the CER trend in different sample datastore size.