Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DuplexCascade: Full-Duplex Speech-to-Speech Dialogue with VAD-Free Cascaded ASR-LLM-TTS Pipeline and Micro-Turn Optimization

Jianing Yang, Yusuke Fujita, Yui Sudo

TL;DR

The key idea is to convert conventional utterance-wise long turns into chunk-wise micro-turn interactions, enabling rapid bidirectional exchange while preserving the strengths of a capable text LLM, and introduce a set of conversational special control tokens that steer the LLM's behavior under streaming constraints.

Abstract

Spoken dialog systems with cascaded ASR-LLM-TTS modules retain strong LLM intelligence, but VAD segmentation often forces half-duplex turns and brittle control. On the other hand, VAD-free end-to-end model support full-duplex interaction but is hard to maintain conversational intelligence. In this paper, we present DuplexCascade, a VAD-free cascaded streaming pipeline for full-duplex speech-to-speech dialogue. Our key idea is to convert conventional utterance-wise long turns into chunk-wise micro-turn interactions, enabling rapid bidirectional exchange while preserving the strengths of a capable text LLM. To reliably coordinate turn-taking and response timing, we introduce a set of conversational special control tokens that steer the LLM's behavior under streaming constraints. On Full-DuplexBench and VoiceBench, DuplexCascade delivers state-of-the-art full-duplex turn-taking and strong conversational intelligence among open-source speech-to-speech dialogue systems.

DuplexCascade: Full-Duplex Speech-to-Speech Dialogue with VAD-Free Cascaded ASR-LLM-TTS Pipeline and Micro-Turn Optimization

TL;DR

The key idea is to convert conventional utterance-wise long turns into chunk-wise micro-turn interactions, enabling rapid bidirectional exchange while preserving the strengths of a capable text LLM, and introduce a set of conversational special control tokens that steer the LLM's behavior under streaming constraints.

Abstract

Spoken dialog systems with cascaded ASR-LLM-TTS modules retain strong LLM intelligence, but VAD segmentation often forces half-duplex turns and brittle control. On the other hand, VAD-free end-to-end model support full-duplex interaction but is hard to maintain conversational intelligence. In this paper, we present DuplexCascade, a VAD-free cascaded streaming pipeline for full-duplex speech-to-speech dialogue. Our key idea is to convert conventional utterance-wise long turns into chunk-wise micro-turn interactions, enabling rapid bidirectional exchange while preserving the strengths of a capable text LLM. To reliably coordinate turn-taking and response timing, we introduce a set of conversational special control tokens that steer the LLM's behavior under streaming constraints. On Full-DuplexBench and VoiceBench, DuplexCascade delivers state-of-the-art full-duplex turn-taking and strong conversational intelligence among open-source speech-to-speech dialogue systems.
Paper Structure (19 sections, 3 figures, 2 tables)

This paper contains 19 sections, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. VAD-Based Turn Taking Control
  4. VAD-Free Time-Division-Multiplexing Approach Toward Full-Duplex Interaction
  5. DuplexCascade
  6. Overview
  7. Conversational Special Tokens
  8. User's special token.
  9. System's special tokens.
  10. Dynamic Construction of Duplex Training Data
  11. Convert text dialogue into micro-turn segmentation
  12. Simulating duplex interaction phenomena
  13. Experiments
  14. Implementation Details
  15. Full-Duplex-Bench Results
  16. ...and 4 more sections

Figures (3)

  • Figure 1: Overview of DuplexCascade. User audio is transcribed by a streaming ASR and periodically flushed into text micro-turns ($\Delta t=0.6\,$s). The LLM consumes the dialogue history and the latest micro-turn to generate the next system micro-turn together with conversational special tokens (e.g., wait, respond, or backchannel). The generated text is then synthesized by a streaming TTS to produce system audio, enabling full-duplex interaction.
  • Figure 2: Proposed dynamic construction pipeline for duplex training sequences from text-only dialogues.
  • Figure 3: Effect of the micro-turn duration $\Delta t$ on Averaged Turn-Taking Accuracy and turn-taking latency on Full-Duplex-Bench.