Multi-Modal Retrieval For Large Language Model Based Speech Recognition

TL;DR

This paper tackles the problem of improving speech recognition by augmenting multi-modal language models with retrieval from external, audio-informed memory. It introduces two retrieval paradigms—kNN-LM and cross-attention—applied to small and large multi-modal LMs that encode audio using HuBERT-derived tokens, achieving up to 50% relative improvements in word error rate and reaching state-of-the-art results on Spoken-Squad QA. The cross-attention approach generally outperforms kNN-LM, particularly for dynamic-context scenarios, and the work demonstrates practical retrieval latency that is modest relative to decoding time. Overall, the study shows that multi-modal retrieval enhances ASR performance, facilitates domain adaptation, and suggests a path toward more efficient deployment by reusing key encoders across retrieval and generation components.

Abstract

Retrieval is a widely adopted approach for improving language models leveraging external information. As the field moves towards multi-modal large language models, it is important to extend the pure text based methods to incorporate other modalities in retrieval as well for applications across the wide spectrum of machine learning tasks and data types. In this work, we propose multi-modal retrieval with two approaches: kNN-LM and cross-attention techniques. We demonstrate the effectiveness of our retrieval approaches empirically by applying them to automatic speech recognition tasks with access to external information. Under this setting, we show that speech-based multi-modal retrieval outperforms text based retrieval, and yields up to 50 % improvement in word error rate over the multi-modal language model baseline. Furthermore, we achieve state-of-the-art recognition results on the Spoken-Squad question answering dataset.

Figures (3)

• Figure 1: Illustration of a speech multi-modal LM. Inputs to the LM consist of three parts: a prompt specifying the task, audio tokens from an audio tokenizer, and text tokens.
• Figure 2: (a): Illustration of the cross-attention based retrieval model. Input tokens are used as inputs both to the decoder model (shown on left-hand side) and key encoder (shown on the right-hand side). The encoded value are used as inputs to the decoder as the key and query for multi-head cross-attention (shown with the red-block). The depicted transformer architecture (normalization layers, etc.) is for illustration purposes and may vary slightly between different models. (b) Illustration of retrieval database creation. Text tokens are encoded and used as keys for the database. Values are surrounding tokens of the key.
• Figure 3: Illustration of the token level multi-head cross-attention. Here $e^l_{i}$ is the $i^{\mathrm{th}}$ token of the $l^{\mathrm{th}}$ layer. Color highlights the interactions between the context and the query tokens. MHA stands for standard multi-head cross-attention. The dashed arrow lines from the MHA outputs illustrate the causal dependencies.