Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PromptStyle: Controllable Style Transfer for Text-to-Speech with Natural Language Descriptions

Guanghou Liu, Yongmao Zhang, Yi Lei, Yunlin Chen, Rui Wang, Zhifei Li, Lei Xie

TL;DR

The paper tackles flexible, user-friendly style control in TTS by enabling cross-speaker style transfer guided by natural language prompts. It introduces PromptStyle, a three-component system based on VITS, incorporating a cross-modal style encoder and a BERT-powered prompt encoder, trained in two stages to align style and prompt spaces. Stage 1 learns style from reference speech, while Stage 2 links text prompts to the style space, allowing unseen prompts to guide synthesis. Experimental results on audiobook data show that PromptStyle delivers high speech quality, strong speaker similarity, and coherent style transfer with text prompts, outperforming baseline approaches. This work broadens practical TTS customization by enabling intuitive linguistic descriptions to steer expressiveness across speakers.

Abstract

Style transfer TTS has shown impressive performance in recent years. However, style control is often restricted to systems built on expressive speech recordings with discrete style categories. In practical situations, users may be interested in transferring style by typing text descriptions of desired styles, without the reference speech in the target style. The text-guided content generation techniques have drawn wide attention recently. In this work, we explore the possibility of controllable style transfer with natural language descriptions. To this end, we propose PromptStyle, a text prompt-guided cross-speaker style transfer system. Specifically, PromptStyle consists of an improved VITS and a cross-modal style encoder. The cross-modal style encoder constructs a shared space of stylistic and semantic representation through a two-stage training process. Experiments show that PromptStyle can achieve proper style transfer with text prompts while maintaining relatively high stability and speaker similarity. Audio samples are available in our demo page.

PromptStyle: Controllable Style Transfer for Text-to-Speech with Natural Language Descriptions

TL;DR

The paper tackles flexible, user-friendly style control in TTS by enabling cross-speaker style transfer guided by natural language prompts. It introduces PromptStyle, a three-component system based on VITS, incorporating a cross-modal style encoder and a BERT-powered prompt encoder, trained in two stages to align style and prompt spaces. Stage 1 learns style from reference speech, while Stage 2 links text prompts to the style space, allowing unseen prompts to guide synthesis. Experimental results on audiobook data show that PromptStyle delivers high speech quality, strong speaker similarity, and coherent style transfer with text prompts, outperforming baseline approaches. This work broadens practical TTS customization by enabling intuitive linguistic descriptions to steer expressiveness across speakers.

Abstract

Style transfer TTS has shown impressive performance in recent years. However, style control is often restricted to systems built on expressive speech recordings with discrete style categories. In practical situations, users may be interested in transferring style by typing text descriptions of desired styles, without the reference speech in the target style. The text-guided content generation techniques have drawn wide attention recently. In this work, we explore the possibility of controllable style transfer with natural language descriptions. To this end, we propose PromptStyle, a text prompt-guided cross-speaker style transfer system. Specifically, PromptStyle consists of an improved VITS and a cross-modal style encoder. The cross-modal style encoder constructs a shared space of stylistic and semantic representation through a two-stage training process. Experiments show that PromptStyle can achieve proper style transfer with text prompts while maintaining relatively high stability and speaker similarity. Audio samples are available in our demo page.
Paper Structure (16 sections, 2 equations, 3 figures, 3 tables)

This paper contains 16 sections, 2 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. PromptStyle
  3. Model Overview
  4. Cross-modal Style Encoder
  5. Training and Inference
  6. Stage 1: Style transfer by reference Mel
  7. Stage 2: Style transfer by text prompt
  8. Experiments
  9. Datasets
  10. Model Configuration
  11. Performance on the Style Transfer
  12. Subjective Evaluation
  13. Objective evaluation
  14. Ablation Study
  15. Style Control by Text Prompt
  16. ...and 1 more sections

Figures (3)

  • Figure 1: Architecture of PromptStyle
  • Figure 2: Visualization of the style embeddings from different models -- (a) CST-TTS, (b) GST-MLTTS and (c) PromptStyle.
  • Figure 3: ABX preference results