Hearing More with Less: Multi-Modal Retrieval-and-Selection Augmented Conversational LLM-Based ASR

TL;DR

The paper tackles the challenge of leveraging conversational history for LLM-based ASR without incurring excessive computation or being overwhelmed by irrelevant content. It introduces MARS, a multi-modal retrieval-and-selection framework that fetches candidate historical contexts from both speech and text modalities and uses a near-ideal ranking strategy to select the best one to condition the LLM. MARS employs a Whisper-based database, DTW/FastDTW for acoustic similarity, embedding-based text similarity, and joint training of a projector and LoRA to transcribe, achieving strong performance on the Interspeech 2025 MLC-SLM dataset with only 1.5K hours of training data and setting a new state-of-the-art. These results demonstrate efficient data utilization and practical viability of retrieval-guided context augmentation for conversational LLM-ASR, offering a path to high accuracy with much smaller labeled data requirements.

Abstract

Automatic Speech Recognition (ASR) aims to convert human speech content into corresponding text. In conversational scenarios, effectively utilizing context can enhance its accuracy. Large Language Models' (LLMs) exceptional long-context understanding and reasoning abilities enable LLM-based ASR (LLM-ASR) to leverage historical context for recognizing conversational speech, which has a high degree of contextual relevance. However, existing conversational LLM-ASR methods use a fixed number of preceding utterances or the entire conversation history as context, resulting in significant ASR confusion and computational costs due to massive irrelevant and redundant information. This paper proposes a multi-modal retrieval-and-selection method named MARS that augments conversational LLM-ASR by enabling it to retrieve and select the most relevant acoustic and textual historical context for the current utterance. Specifically, multi-modal retrieval obtains a set of candidate historical contexts, each exhibiting high acoustic or textual similarity to the current utterance. Multi-modal selection calculates the acoustic and textual similarities for each retrieved candidate historical context and, by employing our proposed near-ideal ranking method to consider both similarities, selects the best historical context. Evaluations on the Interspeech 2025 Multilingual Conversational Speech Language Model Challenge dataset show that the LLM-ASR, when trained on only 1.5K hours of data and equipped with the MARS, outperforms the state-of-the-art top-ranking system trained on 179K hours of data.

Figures (6)

• Figure 1: Similarity and content of current utterances and historical contexts in conversational speech. The closer the historical context is to red, the more similar it is to the current utterance. The purple "Context{N}" represents the most relevant historical context, and "Context{1$\sim$N} refers to the preceding N contexts.
• Figure 2: Overview of MARS. Left: a language-aware prompt, the best retrieved historical context, the current utterance speech embedding, and its hypothesis are used as inputs to the LLM for jointly training the projector and LoRA parameters to predict the transcription; Middle: the multi-modal retrieval module retrieves historical contexts from the database based on speech and text queries, while the multi-modal selection module determines the best historical context from the retrieved ones; Right: the database is constructed using Whisper, storing a triplet for each utterance in the conversation.
• Figure 3: The details of the multi-modal retrieval module.
• Figure 4: The details of the multi-modal selection module.
• Figure 5: Average speech and text retrieval similarity between different types of historical contexts and the current utterance in the MLC-SLM test dataset.