Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SpeakStream: Streaming Text-to-Speech with Interleaved Data

Richard He Bai, Zijin Gu, Tatiana Likhomanenko, Navdeep Jaitly

TL;DR

SpeakStream addresses high first-token latency in cascaded LLM+TTS systems by introducing a decoder-only streaming TTS that interleaves text and speech into a single sequence and is trained with next-step prediction on interleaved data. It eliminates explicit alignment, uses kv-cache for efficiency, and pairs with VocStream to deliver end-to-end streaming with first-frame latency around $30\,\mathrm{ms}$ and total latency under $50\,\mathrm{ms}$ on a Mac Mini. Empirical results show competitive WER and human evaluations indicate maintained coherence compared with non-streaming baselines, demonstrating practical viability for interactive agents. The work highlights a unified, low-latency approach to streaming TTS that could underpin responsive conversational systems and cascaded AI pipelines.

Abstract

The latency bottleneck of traditional text-to-speech (TTS) systems fundamentally hinders the potential of streaming large language models (LLMs) in conversational AI. These TTS systems, typically trained and inferenced on complete utterances, introduce unacceptable delays, even with optimized inference speeds, when coupled with streaming LLM outputs. This is particularly problematic for creating responsive conversational agents where low first-token latency is critical. In this paper, we present SpeakStream, a streaming TTS system that generates audio incrementally from streaming text using a decoder-only architecture. SpeakStream is trained using a next-step prediction loss on interleaved text-speech data. During inference, it generates speech incrementally while absorbing streaming input text, making it particularly suitable for cascaded conversational AI agents where an LLM streams text to a TTS system. Our experiments demonstrate that SpeakStream achieves state-of-the-art latency results in terms of first-token latency while maintaining the quality of non-streaming TTS systems.

SpeakStream: Streaming Text-to-Speech with Interleaved Data

TL;DR

SpeakStream addresses high first-token latency in cascaded LLM+TTS systems by introducing a decoder-only streaming TTS that interleaves text and speech into a single sequence and is trained with next-step prediction on interleaved data. It eliminates explicit alignment, uses kv-cache for efficiency, and pairs with VocStream to deliver end-to-end streaming with first-frame latency around and total latency under on a Mac Mini. Empirical results show competitive WER and human evaluations indicate maintained coherence compared with non-streaming baselines, demonstrating practical viability for interactive agents. The work highlights a unified, low-latency approach to streaming TTS that could underpin responsive conversational systems and cascaded AI pipelines.

Abstract

The latency bottleneck of traditional text-to-speech (TTS) systems fundamentally hinders the potential of streaming large language models (LLMs) in conversational AI. These TTS systems, typically trained and inferenced on complete utterances, introduce unacceptable delays, even with optimized inference speeds, when coupled with streaming LLM outputs. This is particularly problematic for creating responsive conversational agents where low first-token latency is critical. In this paper, we present SpeakStream, a streaming TTS system that generates audio incrementally from streaming text using a decoder-only architecture. SpeakStream is trained using a next-step prediction loss on interleaved text-speech data. During inference, it generates speech incrementally while absorbing streaming input text, making it particularly suitable for cascaded conversational AI agents where an LLM streams text to a TTS system. Our experiments demonstrate that SpeakStream achieves state-of-the-art latency results in terms of first-token latency while maintaining the quality of non-streaming TTS systems.

Paper Structure

This paper contains 14 sections, 6 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Text and Speech Representation
  5. SpeakStream Model
  6. Interleaving Schemes
  7. Streaming Inference
  8. VocStream: Streaming Vocoder for Streaming TTS
  9. Experiments
  10. Setup
  11. Main Results
  12. Human Evaluation
  13. Latency Analysis
  14. Conclusion

Figures (3)

  • Figure 1: First-token to speech (time taken from the moment the LLM produces the first text token, until the TTS produces the first byte of speech) latency comparison between SpeakStream and other baselines. The 45ms latency is tested with MLX mlx2023 and Mac Mini with M4 Pro chip and 64GB. Baselines latency is tested from https://github.com/Picovoice/tts-latency-benchmark.
  • Figure 2: SpeakStream transformer decoder architecture.
  • Figure 3: A 30-second Mel-spectrogram generated from the SpeakStream model. It corresponds to the audio in our demo, whose transcription is the abstract section of this paper.