Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Discourse-Aware Dual-Track Streaming Response for Low-Latency Spoken Dialogue Systems

Siyuan Liu, Jiahui Xu, Feng Jiang, Kuang Wang, Zefeng Zhao, Chu-Ren Huang, Jinghang Gu, Changqing Yin, Haizhou Li

TL;DR

Analysis shows that DDTSR functions as a plug-and-play module compatible with diverse LLM backbones, and remains robust across varying utterance lengths, indicating strong practicality and scalability for real-time spoken interaction.

Abstract

Achieving human-like responsiveness is a critical yet challenging goal for cascaded spoken dialogue systems. Conventional ASR-LLM-TTS pipelines follow a strictly sequential paradigm, requiring complete transcription and full reasoning before speech synthesis can begin, which results in high response latency. We propose the Discourse-Aware Dual-Track Streaming Response (DDTSR) framework, a low-latency architecture that enables listen-while-thinking and speak-while-thinking. DDTSR is built upon three key mechanisms: (1) connective-guided small-large model synergy, where an auxiliary small model generates minimal-committal discourse connectives while a large model performs knowledge-intensive reasoning in parallel; (2) streaming-based cross-modal collaboration, which dynamically overlaps ASR, LLM inference, and TTS to advance the earliest speakable moment; and (3) curriculum-learning-based discourse continuity enhancement, which maintains coherence and logical consistency between early responses and subsequent reasoning outputs. Experiments on two spoken dialogue benchmarks demonstrate that DDTSR reduces response latency by 19%-51% while preserving discourse quality. Further analysis shows that DDTSR functions as a plug-and-play module compatible with diverse LLM backbones, and remains robust across varying utterance lengths, indicating strong practicality and scalability for real-time spoken interaction.

Discourse-Aware Dual-Track Streaming Response for Low-Latency Spoken Dialogue Systems

TL;DR

Analysis shows that DDTSR functions as a plug-and-play module compatible with diverse LLM backbones, and remains robust across varying utterance lengths, indicating strong practicality and scalability for real-time spoken interaction.

Abstract

Achieving human-like responsiveness is a critical yet challenging goal for cascaded spoken dialogue systems. Conventional ASR-LLM-TTS pipelines follow a strictly sequential paradigm, requiring complete transcription and full reasoning before speech synthesis can begin, which results in high response latency. We propose the Discourse-Aware Dual-Track Streaming Response (DDTSR) framework, a low-latency architecture that enables listen-while-thinking and speak-while-thinking. DDTSR is built upon three key mechanisms: (1) connective-guided small-large model synergy, where an auxiliary small model generates minimal-committal discourse connectives while a large model performs knowledge-intensive reasoning in parallel; (2) streaming-based cross-modal collaboration, which dynamically overlaps ASR, LLM inference, and TTS to advance the earliest speakable moment; and (3) curriculum-learning-based discourse continuity enhancement, which maintains coherence and logical consistency between early responses and subsequent reasoning outputs. Experiments on two spoken dialogue benchmarks demonstrate that DDTSR reduces response latency by 19%-51% while preserving discourse quality. Further analysis shows that DDTSR functions as a plug-and-play module compatible with diverse LLM backbones, and remains robust across varying utterance lengths, indicating strong practicality and scalability for real-time spoken interaction.
Paper Structure (42 sections, 12 equations, 5 figures, 6 tables, 1 algorithm)

This paper contains 42 sections, 12 equations, 5 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Cascaded Spoken Dialogue System
  4. Latency Reduction in Cascaded Systems
  5. Faster computation is not faster interaction.
  6. Early response and incremental generation.
  7. Discourse-level Cues for Conversational Naturalness
  8. Discourse-Aware Dual-Track Streaming Response Framework (DDTSR)
  9. Connective-Guided Small--Large Model Synergy
  10. Streaming-Based Cross-Modal Collaboration
  11. (1) Early emission during streaming ASR.
  12. (2) Parallel small–large model execution across modalities.
  13. Curriculum-Learning-Based Discourse Continuity Enhancement
  14. (1) Discourse-oriented joint training.
  15. (2) Curriculum-based optimization.
  16. ...and 27 more sections

Figures (5)

  • Figure 1: Necessity of low-latency spoken interaction. Humans naturally exhibit speaking while thinking behavior, whereas conventional systems wait for full response completion before speaking.
  • Figure 2: The Framework of Discourse-Aware Dual-Track Streaming Response.
  • Figure 3: Latency analysis on SD-Eval with respect to input audio length.
  • Figure 4: LLM prompt format for turn-transition connective generation.
  • Figure : POS-Based connective extraction pipeline.