MPE-TTS: Customized Emotion Zero-Shot Text-To-Speech Using Multi-Modal Prompt
Zhichao Wu, Yueteng Kang, Songjun Cao, Long Ma, Qiulin Li, Qun Yang
TL;DR
The paper tackles zero-shot TTS by enabling flexible, fine-grained emotional control through multi-modal prompts. It presents MPE-TTS, featuring a hierarchical disentangling strategy, a multi-modal prompt emotion encoder based on Emotion2Vec, an LLM-like prosody predictor, and a diffusion-based acoustic model to synthesize speech from text, image, or speech prompts. Key contributions include the MPEE loss that unifies cross-modal emotion representations and the Emotion Consistency Loss that preserves emotion information in predicted prosody. Experiments on LibriTTS and MEAD-TTS demonstrate improved naturalness and emotion similarity, with strong performance across different prompt modalities and evidence from ablation studies.
Abstract
Most existing Zero-Shot Text-To-Speech(ZS-TTS) systems generate the unseen speech based on single prompt, such as reference speech or text descriptions, which limits their flexibility. We propose a customized emotion ZS-TTS system based on multi-modal prompt. The system disentangles speech into the content, timbre, emotion and prosody, allowing emotion prompts to be provided as text, image or speech. To extract emotion information from different prompts, we propose a multi-modal prompt emotion encoder. Additionally, we introduce an prosody predictor to fit the distribution of prosody and propose an emotion consistency loss to preserve emotion information in the predicted prosody. A diffusion-based acoustic model is employed to generate the target mel-spectrogram. Both objective and subjective experiments demonstrate that our system outperforms existing systems in terms of naturalness and similarity. The samples are available at https://mpetts-demo.github.io/mpetts_demo/.