Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ESPnet-SDS: Unified Toolkit and Demo for Spoken Dialogue Systems

Siddhant Arora, Yifan Peng, Jiatong Shi, Jinchuan Tian, William Chen, Shikhar Bharadwaj, Hayato Futami, Yosuke Kashiwagi, Emiru Tsunoo, Shuichiro Shimizu, Vaibhav Srivastav, Shinji Watanabe

TL;DR

ESPnet-SDS delivers an open-source, unified Gradio-based toolkit to compare cascaded and end-to-end spoken dialogue systems under a single interface. It provides on-the-fly evaluation across ASR, TTS, dialogue generation, and conversation-level metrics, plus human-in-the-loop feedback, enabling reproducible benchmarking on Switchboard Eval 2000. The initial analysis shows current E2E models often lag in audio quality and response diversity compared with cascaded approaches, yet the platform enables actionable design insights and rapid experimentation. By offering extensible wrappers and templates, ESPnet-SDS aims to accelerate robust development and fair comparison of SDS technologies in a research and production context.

Abstract

Advancements in audio foundation models (FMs) have fueled interest in end-to-end (E2E) spoken dialogue systems, but different web interfaces for each system makes it challenging to compare and contrast them effectively. Motivated by this, we introduce an open-source, user-friendly toolkit designed to build unified web interfaces for various cascaded and E2E spoken dialogue systems. Our demo further provides users with the option to get on-the-fly automated evaluation metrics such as (1) latency, (2) ability to understand user input, (3) coherence, diversity, and relevance of system response, and (4) intelligibility and audio quality of system output. Using the evaluation metrics, we compare various cascaded and E2E spoken dialogue systems with a human-human conversation dataset as a proxy. Our analysis demonstrates that the toolkit allows researchers to effortlessly compare and contrast different technologies, providing valuable insights such as current E2E systems having poorer audio quality and less diverse responses. An example demo produced using our toolkit is publicly available here: https://huggingface.co/spaces/Siddhant/Voice_Assistant_Demo.

ESPnet-SDS: Unified Toolkit and Demo for Spoken Dialogue Systems

TL;DR

ESPnet-SDS delivers an open-source, unified Gradio-based toolkit to compare cascaded and end-to-end spoken dialogue systems under a single interface. It provides on-the-fly evaluation across ASR, TTS, dialogue generation, and conversation-level metrics, plus human-in-the-loop feedback, enabling reproducible benchmarking on Switchboard Eval 2000. The initial analysis shows current E2E models often lag in audio quality and response diversity compared with cascaded approaches, yet the platform enables actionable design insights and rapid experimentation. By offering extensible wrappers and templates, ESPnet-SDS aims to accelerate robust development and fair comparison of SDS technologies in a research and production context.

Abstract

Advancements in audio foundation models (FMs) have fueled interest in end-to-end (E2E) spoken dialogue systems, but different web interfaces for each system makes it challenging to compare and contrast them effectively. Motivated by this, we introduce an open-source, user-friendly toolkit designed to build unified web interfaces for various cascaded and E2E spoken dialogue systems. Our demo further provides users with the option to get on-the-fly automated evaluation metrics such as (1) latency, (2) ability to understand user input, (3) coherence, diversity, and relevance of system response, and (4) intelligibility and audio quality of system output. Using the evaluation metrics, we compare various cascaded and E2E spoken dialogue systems with a human-human conversation dataset as a proxy. Our analysis demonstrates that the toolkit allows researchers to effortlessly compare and contrast different technologies, providing valuable insights such as current E2E systems having poorer audio quality and less diverse responses. An example demo produced using our toolkit is publicly available here: https://huggingface.co/spaces/Siddhant/Voice_Assistant_Demo.

Paper Structure

This paper contains 18 sections, 2 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. System Design
  4. Example Models
  5. Cascaded Models
  6. E2E Spoken Dialogue System
  7. Evaluation Metrics
  8. Web Interface
  9. Analysis using Evaluation Metrics
  10. Conclusion and Outlook
  11. Limitation
  12. Broader Impact and Ethics
  13. Example Appendix
  14. Spoken Dialogue Systems
  15. Evaluation Setup
  16. ...and 3 more sections

Figures (2)

  • Figure 1: Screenshot of our unified web interface highlighting key features: (a) streaming audio input, (b) display of ASR transcripts, text responses, and synthesized audio outputs, (c) interaction with cascaded and E2E dialogue systems and experimentation with ASR, LLM, and TTS submodules, (d) on-the-fly evaluation metrics, and (e) collection of human feedback on the naturalness and relevance of system outputs.
  • Figure 2: Modular software architecture of ESPnet-SDS