DurIAN: Duration Informed Attention Network For Multimodal Synthesis

TL;DR

DurIAN addresses robustness and synchronization in multimodal TTS by substituting end-to-end attention with a duration-informed alignment guided by a duration model. The architecture combines a skip encoder, alignment model, decoder, and post-net, enabling joint speech and facial expression synthesis with optional paired data. A fine-grained style control mechanism allows continuous manipulation of speaking style, while the Multi-band WaveRNN framework significantly speeds up vocoder inference without harming quality. Experimental results show DurIAN achieves naturalness on par with Tacotron-2 and demonstrates robust performance, plus strong speedups for real-time applications and flexible multimodal synchronization without requiring identical training data for speech and faces.

Abstract

In this paper, we present a generic and robust multimodal synthesis system that produces highly natural speech and facial expression simultaneously. The key component of this system is the Duration Informed Attention Network (DurIAN), an autoregressive model in which the alignments between the input text and the output acoustic features are inferred from a duration model. This is different from the end-to-end attention mechanism used, and accounts for various unavoidable artifacts, in existing end-to-end speech synthesis systems such as Tacotron. Furthermore, DurIAN can be used to generate high quality facial expression which can be synchronized with generated speech with/without parallel speech and face data. To improve the efficiency of speech generation, we also propose a multi-band parallel generation strategy on top of the WaveRNN model. The proposed Multi-band WaveRNN effectively reduces the total computational complexity from 9.8 to 5.5 GFLOPS, and is able to generate audio that is 6 times faster than real time on a single CPU core. We show that DurIAN could generate highly natural speech that is on par with current state of the art end-to-end systems, while at the same time avoid word skipping/repeating errors in those systems. Finally, a simple yet effective approach for fine-grained control of expressiveness of speech and facial expression is introduced.

Figures (5)

• Figure 1: Model architecture of DurIAN. The model takes a sequence of symbols, including phonemes and prosodic boundaries between them, and outputs the corresponding mel-spectrogram and facial expression control parameters.
• Figure 2: An illustration of how prosodic boundaries are inserted between input phonemes. The symbol #S represents the boundary of syllables, #1 represents the boundary of prosodic words, #2 represents the boundary of prosodic phrase, and #3 represents the boundary of intonational phrase.
• Figure 3: Generation of the style code from style embedding. The style embedding is learned jointly with other parameters of DurIAN through multi-style training.
• Figure 4: Model architecture of Multi-band WaveRNN.
• Figure 5: Frequency response of 4-band Pseudo-QMF filter banks.