Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Speech-to-Speech Dialogue Modeling with End-to-End Retrieval-Augmented Generation

Pengchao Feng, Ziyang Ma, Wenxi Chen, Yao Li, Sheng Wang, Kai Yu, Xie Chen

TL;DR

This work tackles end-to-end speech-to-speech dialogue by introducing a retrieval-augmented generation framework that directly retrieves textual knowledge from speech through a shared embedding space. Leveraging a multimodal encoder (SONAR), the approach eliminates ASR bottlenecks to achieve significant retrieval speedups while preserving substantial knowledge grounding. Empirical results on multilingual QA benchmarks show meaningful improvements over non-RAG baselines, though the end-to-end method still trails ASR-based cascades in some metrics. Ablation studies emphasize the importance of semantic alignment in the shared space, and the authors release code and data to encourage further research in this direction.

Abstract

End-to-end speech-to-speech (S2S) dialogue systems have recently garnered increasing research attention for their lower latency and more natural integration of nonverbal cues such as emotion and speaker identity. However, these systems face key challenges, particularly in incorporating external knowledge, a capability commonly addressed by Retrieval-Augmented Generation (RAG) in text-based large language models (LLMs). The core difficulty lies in the modality gap between input speech and retrieved textual knowledge, which hinders effective integration of information. To address this issue, we propose a novel end-to-end RAG framework that directly retrieves relevant textual knowledge from speech queries. Experimental results demonstrate that our method significantly improves the performance of end-to-end S2S dialogue systems while achieving higher retrieval efficiency. Although the overall performance still lags behind the SOTA cascaded models, our framework offers a promising direction for enhancing knowledge integration in end-to-end S2S systems. Our code and dataset are released.

Enhancing Speech-to-Speech Dialogue Modeling with End-to-End Retrieval-Augmented Generation

TL;DR

This work tackles end-to-end speech-to-speech dialogue by introducing a retrieval-augmented generation framework that directly retrieves textual knowledge from speech through a shared embedding space. Leveraging a multimodal encoder (SONAR), the approach eliminates ASR bottlenecks to achieve significant retrieval speedups while preserving substantial knowledge grounding. Empirical results on multilingual QA benchmarks show meaningful improvements over non-RAG baselines, though the end-to-end method still trails ASR-based cascades in some metrics. Ablation studies emphasize the importance of semantic alignment in the shared space, and the authors release code and data to encourage further research in this direction.

Abstract

End-to-end speech-to-speech (S2S) dialogue systems have recently garnered increasing research attention for their lower latency and more natural integration of nonverbal cues such as emotion and speaker identity. However, these systems face key challenges, particularly in incorporating external knowledge, a capability commonly addressed by Retrieval-Augmented Generation (RAG) in text-based large language models (LLMs). The core difficulty lies in the modality gap between input speech and retrieved textual knowledge, which hinders effective integration of information. To address this issue, we propose a novel end-to-end RAG framework that directly retrieves relevant textual knowledge from speech queries. Experimental results demonstrate that our method significantly improves the performance of end-to-end S2S dialogue systems while achieving higher retrieval efficiency. Although the overall performance still lags behind the SOTA cascaded models, our framework offers a promising direction for enhancing knowledge integration in end-to-end S2S systems. Our code and dataset are released.
Paper Structure (18 sections, 2 equations, 1 figure, 2 tables)

This paper contains 18 sections, 2 equations, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. System overview
  5. Indexing
  6. Retrieval
  7. Generation
  8. Experiments
  9. Dataset
  10. Baseline systems
  11. Result
  12. Ablation study
  13. Conclusion
  14. Appendix
  15. Experiment Config
  16. ...and 3 more sections

Figures (1)

  • Figure 1: Our proposed framework of end-to-end RAG for speech-to-speech dialogue model. 1) Indexing stage, where text information is divided into chunks and encoded index. 2) Retrieval stage, where we compare ASR-based cascade retrieval, which transcribes speech to text before performing text retrieval, and end-to-end retrieval, which encodes speech queries into the same vector space as texts for direct retrieval. 3) Generation stage, where the end-to-end S2S dialogue model generates responses.