Retrieval Augmented End-to-End Spoken Dialog Models

TL;DR

The paper introduces Retrieval Augmented SLM (ReSLM), which adds a speech retriever to a joint speech-language model to fetch domain-specific text entities from audio and concatenate them to the model inputs. Trained with a contrastive dual-encoder retriever and an end-to-end SLM, ReSLM outperforms the baseline SLM on the DSTC11 dialog state tracking task, achieving higher Joint Goal Accuracy and lower Word Error Rate, especially for long-tail entities like hotel, restaurant, and train destinations. The approach directly tackles ASR/NLU errors for domain-specific terms and demonstrates potential applicability to contextual ASR and other speech understanding tasks requiring external knowledge. This work highlights the value of grounding speech models with retrieved textual context to improve accuracy in knowledge-intensive dialog settings.

Abstract

We recently developed SLM, a joint speech and language model, which fuses a pretrained foundational speech model and a large language model (LLM), while preserving the in-context learning capability intrinsic to the pretrained LLM. In this paper, we apply SLM to speech dialog applications where the dialog states are inferred directly from the audio signal. Task-oriented dialogs often contain domain-specific entities, i.e., restaurants, hotels, train stations, and city names, which are difficult to recognize, however, critical for the downstream applications. Inspired by the RAG (retrieval-augmented generation) paradigm, we propose a retrieval augmented SLM (ReSLM) that overcomes this weakness. We first train a speech retriever to retrieve text entities mentioned in the audio. The retrieved entities are then added as text inputs to the underlying SLM to bias model predictions. We evaluated ReSLM on speech MultiWoz task (DSTC-11 challenge), and found that this retrieval augmentation boosts model performance, achieving joint goal accuracy (38.6% vs 32.7%), slot error rate (20.6% vs 24.8%) and ASR word error rate (5.5% vs 6.7%). While demonstrated on dialog state tracking, our approach is broadly applicable to other speech tasks requiring contextual information or domain-specific entities, such as contextual ASR with biasing capability.

Figures (2)

• Figure 1: ReSLM architecture: a speech retriever (trained separately) is added onto SLM, which can provide additional text input as contextual information to SLM. Using the reduced encodings from the speech encoder as queries, the retriever finds the top-k nearest entities (text) and the entities are concatenated to the text inputs entering SLM.
• Figure 2: Direct speech to dialog state prediction in multi-turn dialogs. Given the speech of the current user turn $i$ and a text transcript of the dialog history, SLM or ReSLM generates a single output sequence for both the corresponding transcript [ASR] and dialog state [DST]. The ASR transcript predicted from the turn $i$ is used as history for the turn $i+1$ in an auto-regressive manner.