Delayed Fusion: Integrating Large Language Models into First-Pass Decoding in End-to-end Speech Recognition

TL;DR

This paper tackles the challenge of incorporating large language models into end-to-end ASR without retraining or excessive computation. It introduces delayed fusion, a decoding-time mechanism that applies LM scores to pruned ASR hypotheses and, when needed, re-tokenizes words to align ASR and LLM vocabularies, enabling efficient and flexible integration. Two fusion schemes—shortest-hypothesis and fixed-interval—pair with batched LLM scoring and GPU KV caching to control the trade-off between speed and accuracy. Experiments on LibriHeavy with OpenLLaMA and Mistral LLMs show that delayed fusion yields faster decoding than shallow fusion while delivering comparable or better WER, and superior performance to N-best rescoring in many settings, supporting streaming deployment and broad applicability.

Abstract

This paper presents an efficient decoding approach for end-to-end automatic speech recognition (E2E-ASR) with large language models (LLMs). Although shallow fusion is the most common approach to incorporate language models into E2E-ASR decoding, we face two practical problems with LLMs. (1) LLM inference is computationally costly. (2) There may be a vocabulary mismatch between the ASR model and the LLM. To resolve this mismatch, we need to retrain the ASR model and/or the LLM, which is at best time-consuming and in many cases not feasible. We propose "delayed fusion," which applies LLM scores to ASR hypotheses with a delay during decoding and enables easier use of pre-trained LLMs in ASR tasks. This method can reduce not only the number of hypotheses scored by the LLM but also the number of LLM inference calls. It also allows re-tokenizion of ASR hypotheses during decoding if ASR and LLM employ different tokenizations. We demonstrate that delayed fusion provides improved decoding speed and accuracy compared to shallow fusion and N-best rescoring using the LibriHeavy ASR corpus and three public LLMs, OpenLLaMA 3B & 7B and Mistral 7B.

Figures (3)

• Figure 1: Re-tokenization for delayed LLM fusion.
• Figure 2: LLM score computation in decoding.
• Figure 3: Delayed fusion vs. $N$-best rescoring for different beam sizes 5, 10, 15, and 20. WER and RTF were measured on "lh-other".