Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Emphasis Rendering for Conversational Text-to-Speech with Multi-modal Multi-scale Context Modeling

Rui Liu, Zhenqi Jia, Jie Yang, Yifan Hu, Haizhou Li

TL;DR

A novel Emphasis Rendering scheme for the CTTS model, termed ER-CTTS, that includes two main components that simultaneously take into account textual and acoustic contexts, and deeply integrate multi-modal and multi-scale context to learn the influence of context on the emphasis expression of the current utterance.

Abstract

Conversational Text-to-Speech (CTTS) aims to accurately express an utterance with the appropriate style within a conversational setting, which attracts more attention nowadays. While recognizing the significance of the CTTS task, prior studies have not thoroughly investigated speech emphasis expression, which is essential for conveying the underlying intention and attitude in human-machine interaction scenarios, due to the scarcity of conversational emphasis datasets and the difficulty in context understanding. In this paper, we propose a novel Emphasis Rendering scheme for the CTTS model, termed ER-CTTS, that includes two main components: 1) we simultaneously take into account textual and acoustic contexts, with both global and local semantic modeling to understand the conversation context comprehensively; 2) we deeply integrate multi-modal and multi-scale context to learn the influence of context on the emphasis expression of the current utterance. Finally, the inferred emphasis feature is fed into the neural speech synthesizer to generate conversational speech. To address data scarcity, we create emphasis intensity annotations on the existing conversational dataset (DailyTalk). Both objective and subjective evaluations suggest that our model outperforms the baseline models in emphasis rendering within a conversational setting. The code and audio samples are available at https://github.com/CodeStoreTTS/ER-CTTS.

Emphasis Rendering for Conversational Text-to-Speech with Multi-modal Multi-scale Context Modeling

TL;DR

A novel Emphasis Rendering scheme for the CTTS model, termed ER-CTTS, that includes two main components that simultaneously take into account textual and acoustic contexts, and deeply integrate multi-modal and multi-scale context to learn the influence of context on the emphasis expression of the current utterance.

Abstract

Conversational Text-to-Speech (CTTS) aims to accurately express an utterance with the appropriate style within a conversational setting, which attracts more attention nowadays. While recognizing the significance of the CTTS task, prior studies have not thoroughly investigated speech emphasis expression, which is essential for conveying the underlying intention and attitude in human-machine interaction scenarios, due to the scarcity of conversational emphasis datasets and the difficulty in context understanding. In this paper, we propose a novel Emphasis Rendering scheme for the CTTS model, termed ER-CTTS, that includes two main components: 1) we simultaneously take into account textual and acoustic contexts, with both global and local semantic modeling to understand the conversation context comprehensively; 2) we deeply integrate multi-modal and multi-scale context to learn the influence of context on the emphasis expression of the current utterance. Finally, the inferred emphasis feature is fed into the neural speech synthesizer to generate conversational speech. To address data scarcity, we create emphasis intensity annotations on the existing conversational dataset (DailyTalk). Both objective and subjective evaluations suggest that our model outperforms the baseline models in emphasis rendering within a conversational setting. The code and audio samples are available at https://github.com/CodeStoreTTS/ER-CTTS.

Paper Structure

This paper contains 36 sections, 3 equations, 5 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Conversational Text-to-Speech
  4. Conversational Dataset
  5. ER-CTTS: Methodology
  6. Task Definition
  7. Model Overview
  8. Text Encoder
  9. Textual Context Encoder
  10. CTE
  11. MFTE
  12. Acoustic Context Encoder
  13. CAE
  14. MFAE
  15. Context Fusion Encoder
  16. ...and 21 more sections

Figures (5)

  • Figure 1: The overview of ER-CTTS, that consists of Text Encoder, Textual Context Encoder, Acoustic Context Encoder, Context Fusion Encoder and the Speech Synthesizer.
  • Figure 2: The detailed diagrams of Memory-enhanced Fine-grained Textual Encoder and Memory-enhanced Fine-grained Acoustic Encoder.
  • Figure 3: Conversation-based Emphasis Annotation Scheme.
  • Figure 4: The annotation process of the Emp-DailyTalk.
  • Figure 5: The mel-spectrogram and F0 plots of synthesized speech from different TTS systems. (The pitch value was separately computed. Blue boxes indicate annotated emphasis positions. The red font indicates the top F0 of the emphasized words that represent the emphasis intensity.)