SpeechWeave: Diverse Multilingual Synthetic Text & Audio Data Generation Pipeline for Training Text to Speech Models

TL;DR

SpeechWeave tackles the data scarcity and licensing barriers in domain-specific, multilingual TTS by delivering an end-to-end synthetic data pipeline that jointly generates diverse text scripts and speaker-standardized audio with at-source normalization. The approach leverages keyphrase sampling, an entity sampler with deterministic normalization, and a text script generator, followed by cross-lingual voice cloning to produce consistent speech across languages. Evaluation shows improved lexical and phonetic diversity (higher diphone coverage and lower mean pairwise similarity) and superior normalization accuracy (0.97 English, 0.94 Spanish) compared to baselines and post-processing normalizers, with strong downstream gains in WER when fine-tuning StyleTTS2. The work demonstrates scalable, licensing-friendly data generation for TTS, with implications for rapid domain adaptation and normalization model training, while acknowledging language and semiotic class limitations and outlining directions for broader language support and styling enhancements.

Abstract

High-quality Text-to-Speech (TTS) model training requires extensive and diverse text and speech data. It is challenging to procure such data from real sources due to issues of domain specificity, licensing, and scalability. Large language models (LLMs) can certainly generate textual data, but they create repetitive text with insufficient variation in the prompt during the generation process. Another important aspect in TTS training data is text normalization. Tools for normalization might occasionally introduce anomalies or overlook valuable patterns, and thus impact data quality. Furthermore, it is also impractical to rely on voice artists for large scale speech recording in commercial TTS systems with standardized voices. To address these challenges, we propose SpeechWeave, a synthetic speech data generation pipeline that is capable of automating the generation of multilingual, domain-specific datasets for training TTS models. Our experiments reveal that our pipeline generates data that is 10-48% more diverse than the baseline across various linguistic and phonetic metrics, along with speaker-standardized speech audio while generating approximately 97% correctly normalized text. Our approach enables scalable, high-quality data generation for TTS training, improving diversity, normalization, and voice consistency in the generated datasets.

Figures (8)

• Figure 1: High-level depiction of a TTS system incorporating normalization
• Figure 2: High-level description of our synthetic text and audio generation pipeline
• Figure 3: Multistep Keyphrase Sampling Pipeline with De-duplication and Keyphrase Store
• Figure 4: Example output from keyphrase sampling pipeline at each step of the prompt chain
• Figure 5: Recipes for generating different semiotic classes and their normalized forms