Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RelayS2S: A Dual-Path Speculative Generation for Real-Time Dialogue

Long Mai

Abstract

Real-time spoken dialogue systems face a fundamental tension between latency and response quality. End-to-end speech-to-speech (S2S) models respond immediately and naturally handle turn-taking, backchanneling, and interruption, but produce semantically weaker outputs. Cascaded pipelines (ASR -> LLM) deliver stronger responses at the cost of latency that grows with model size. We present RelayS2S, a hybrid architecture that runs two paths in parallel upon turn detection. The fast path -- a duplex S2S model -- speculatively drafts a short response prefix that is streamed immediately to TTS for low-latency audio onset, while continuing to monitor live audio events. The slow path -- a cascaded ASR -> LLM pipeline -- generates a higher-quality continuation conditioned on the committed prefix, producing a seamless utterance. A lightweight learned verifier gates the handoff, committing the prefix when appropriate or falling back gracefully to the slow path alone. Experiments show that RelayS2S achieves P90 onset latency comparable to the S2S model while retaining 99% cascaded response quality in average score, with benefits growing as the slow-path model scales. Because the prefix handoff requires no architectural modification to either component, RelayS2S serves as a lightweight, drop-in addition to existing cascaded pipelines. Our code and data are publicly available at: https://github.com/mailong25/relays2s

RelayS2S: A Dual-Path Speculative Generation for Real-Time Dialogue

Abstract

Real-time spoken dialogue systems face a fundamental tension between latency and response quality. End-to-end speech-to-speech (S2S) models respond immediately and naturally handle turn-taking, backchanneling, and interruption, but produce semantically weaker outputs. Cascaded pipelines (ASR -> LLM) deliver stronger responses at the cost of latency that grows with model size. We present RelayS2S, a hybrid architecture that runs two paths in parallel upon turn detection. The fast path -- a duplex S2S model -- speculatively drafts a short response prefix that is streamed immediately to TTS for low-latency audio onset, while continuing to monitor live audio events. The slow path -- a cascaded ASR -> LLM pipeline -- generates a higher-quality continuation conditioned on the committed prefix, producing a seamless utterance. A lightweight learned verifier gates the handoff, committing the prefix when appropriate or falling back gracefully to the slow path alone. Experiments show that RelayS2S achieves P90 onset latency comparable to the S2S model while retaining 99% cascaded response quality in average score, with benefits growing as the slow-path model scales. Because the prefix handoff requires no architectural modification to either component, RelayS2S serves as a lightweight, drop-in addition to existing cascaded pipelines. Our code and data are publicly available at: https://github.com/mailong25/relays2s
Paper Structure (51 sections, 6 equations, 3 figures, 5 tables)

This paper contains 51 sections, 6 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. RelayS2S
  3. Overview
  4. Fast Path: Duplex S2S Model
  5. Streaming speech encoder.
  6. LLM adapter.
  7. LLM backbone.
  8. Forked Speculative Generation
  9. Main online stream.
  10. Speculative stream.
  11. Interruption handling.
  12. Selective Prefix Handoff
  13. Verifier input representation.
  14. Verifier architecture.
  15. Training.
  16. ...and 36 more sections

Figures (3)

  • Figure 1: Inference-time architecture of RelayS2S. The fast path (green) speculatively drafts a response prefix that, if committed by the verifier, is streamed immediately to TTS. The slow path (brown) generates a higher-quality continuation conditioned on the committed prefix, or a full response on fallback.
  • Figure 2: Fast-path duplex S2S model. Speech and agent token embeddings are fused via element-wise addition at each 160 ms time step and passed through the LLM to predict the next control or text token.
  • Figure 3: Training examples for duplex control tokens. Top: the agent emits [STP] when the user interrupts. Bottom: the agent emits [BOC] to produce a backchannel during user speech.