Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Prosodic Boundary-Aware Streaming Generation for LLM-Based TTS with Streaming Text Input

Changsong Liu, Tianrui Wang, Ye Ni, Yizhou Peng, Eng Siong Chng

TL;DR

A prosodic-boundary-aware post-training TTS model is proposed, adapting a pretrained LLM-based TTS model using weakly time-aligned data to learn early stopping at specified content boundaries when provided with limited future text.

Abstract

Streaming TTS that receives streaming text is essential for interactive systems, yet this scheme faces two major challenges: unnatural prosody due to missing lookahead and long-form collapse due to unbounded context. We propose a prosodic-boundary-aware post-training strategy, adapting a pretrained LLM-based TTS model using weakly time-aligned data. Specifically, the model is adapted to learn early stopping at specified content boundaries when provided with limited future text. During inference, a sliding-window prompt carries forward previous text and speech tokens, ensuring bounded context and seamless concatenation. Evaluations show our method outperforms CosyVoice-Style interleaved baseline in both short and long-form scenarios. In long-text synthesis, especially, it achieves a 66.2% absolute reduction in word error rate (from 71.0% to 4.8%) and increases speaker and emotion similarity by 16.1% and 1.5% relatively, offering a robust solution for streaming TTS with incremental text.

Prosodic Boundary-Aware Streaming Generation for LLM-Based TTS with Streaming Text Input

TL;DR

A prosodic-boundary-aware post-training TTS model is proposed, adapting a pretrained LLM-based TTS model using weakly time-aligned data to learn early stopping at specified content boundaries when provided with limited future text.

Abstract

Streaming TTS that receives streaming text is essential for interactive systems, yet this scheme faces two major challenges: unnatural prosody due to missing lookahead and long-form collapse due to unbounded context. We propose a prosodic-boundary-aware post-training strategy, adapting a pretrained LLM-based TTS model using weakly time-aligned data. Specifically, the model is adapted to learn early stopping at specified content boundaries when provided with limited future text. During inference, a sliding-window prompt carries forward previous text and speech tokens, ensuring bounded context and seamless concatenation. Evaluations show our method outperforms CosyVoice-Style interleaved baseline in both short and long-form scenarios. In long-text synthesis, especially, it achieves a 66.2% absolute reduction in word error rate (from 71.0% to 4.8%) and increases speaker and emotion similarity by 16.1% and 1.5% relatively, offering a robust solution for streaming TTS with incremental text.
Paper Structure (20 sections, 4 equations, 2 figures, 3 tables)

This paper contains 20 sections, 4 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Prosodic-Boundary Marker
  4. Training with Weakly Time-Aligned Supervision
  5. Bounded Context and Sliding-Window Continuation
  6. Experiments
  7. Training Datasets
  8. Evaluation Datasets
  9. Evaluation Metrics
  10. Experimental Setup and Baselines
  11. Base Architecture
  12. Baselines and Proposed Method
  13. Results and Discussion
  14. Streaming Latency and Efficiency
  15. Synthesis Quality and Linguistic Fidelity
  16. ...and 5 more sections

Figures (2)

  • Figure 1: The proposed fine-tuning and inferencing pipeline for prosodic boundary-aware streaming LLM-based TTS generation.
  • Figure 2: Ablation study on quality evaluation with different combinations of chunk size k and lookahead context f.