MM-TTS: Multi-modal Prompt based Style Transfer for Expressive Text-to-Speech Synthesis

TL;DR

This work addresses flexible, cross-modal style transfer in expressive TTS by introducing MM-TTS, a framework that unifies multi-modal prompts through an Aligned Multi-Modal Prompt Encoder (AMPE) and transfers style with Style Adaptive Convolutions (SAConv). A Rectified Flow based Refiner further enhances Mel-spectrogram fidelity, forming a two-stage training process. To enable evaluation of multi-modal prompts, the authors construct MEAD-TTS, a dataset pairing speech, face imagery, and text descriptions. Experimental results across reference speech, face, and text description prompts show MM-TTS superior performance in both objective metrics and human judgments, demonstrating effective cross-modal style transfer and generalization to out-of-domain data.

Abstract

The style transfer task in Text-to-Speech refers to the process of transferring style information into text content to generate corresponding speech with a specific style. However, most existing style transfer approaches are either based on fixed emotional labels or reference speech clips, which cannot achieve flexible style transfer. Recently, some methods have adopted text descriptions to guide style transfer. In this paper, we propose a more flexible multi-modal and style controllable TTS framework named MM-TTS. It can utilize any modality as the prompt in unified multi-modal prompt space, including reference speech, emotional facial images, and text descriptions, to control the style of the generated speech in a system. The challenges of modeling such a multi-modal style controllable TTS mainly lie in two aspects:1)aligning the multi-modal information into a unified style space to enable the input of arbitrary modality as the style prompt in a single system, and 2)efficiently transferring the unified style representation into the given text content, thereby empowering the ability to generate prompt style-related voice. To address these problems, we propose an aligned multi-modal prompt encoder that embeds different modalities into a unified style space, supporting style transfer for different modalities. Additionally, we present a new adaptive style transfer method named Style Adaptive Convolutions to achieve a better style representation. Furthermore, we design a Rectified Flow based Refiner to solve the problem of over-smoothing Mel-spectrogram and generate audio of higher fidelity. Since there is no public dataset for multi-modal TTS, we construct a dataset named MEAD-TTS, which is related to the field of expressive talking head. Our experiments on the MEAD-TTS dataset and out-of-domain datasets demonstrate that MM-TTS can achieve satisfactory results based on multi-modal prompts.

Figures (10)

• Figure 1: The general style transfer framework for TTS with multi-modal prompts.
• Figure 2: The model architecture of MM-TTS.
• Figure 3: The architecture of speech style encoder in MM-TTS.
• Figure 4: The data preprocessing pipeline for MM-TTS.
• Figure 5: The text description generation process of the text prompts in MEAD-TTS dataset.