AutoStyle-TTS: Retrieval-Augmented Generation based Automatic Style Matching Text-to-Speech Synthesis
Dan Luo, Chengyuan Ma, Weiqin Li, Jun Wang, Wei Chen, Zhiyong Wu
TL;DR
AutoStyle-TTS addresses prompt selection in TTS by applying retrieval-augmented generation to automatically choose matching speech style prompts from an external knowledge base. The system constructs a Speech Style Knowledge Database and uses a three-embedding scheme (E_profile, E_emotion, E_user) via Llama3.2, PER-LLM-Embedder, and Moka to perform style matching; the TTS core decouples style/timbre and utilizes a CosyVoice-based generation with flow-matching. Experimental results show improved style matching and coherence (MOS) and competitive AB performance against manually selected prompts, validating the practicality of automated style adaptation in podcast-like content. The approach offers a scalable, IR-driven pathway to tailor speech style to content, enabling more natural and expressive TTS across varied contexts.
Abstract
With the advancement of speech synthesis technology, users have higher expectations for the naturalness and expressiveness of synthesized speech. But previous research ignores the importance of prompt selection. This study proposes a text-to-speech (TTS) framework based on Retrieval-Augmented Generation (RAG) technology, which can dynamically adjust the speech style according to the text content to achieve more natural and vivid communication effects. We have constructed a speech style knowledge database containing high-quality speech samples in various contexts and developed a style matching scheme. This scheme uses embeddings, extracted by Llama, PER-LLM-Embedder,and Moka, to match with samples in the knowledge database, selecting the most appropriate speech style for synthesis. Furthermore, our empirical research validates the effectiveness of the proposed method. Our demo can be viewed at: https://thuhcsi.github.io/icme2025-AutoStyle-TTS