Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Perturbation Self-Supervised Representations for Cross-Lingual Emotion TTS: Stage-Wise Modeling of Emotion and Speaker

Cheng Gong, Chunyu Qiang, Tianrui Wang, Yu Jiang, Yuheng Lu, Ruihao Jing, Xiaoxiao Miao, Xiaolei Zhang, Longbiao Wang, Jianwu Dang

TL;DR

The paper tackles cross-lingual emotion TTS, where emotion from a speaker in one language is rendered in another language while preserving timbre. It introduces EMM-TTS, a two-stage framework that leverages perturbed self-supervised representations to decouple emotion from timbre, with stage 1 predicting emotion representations and stage 2 restoring timbre. Key contributions include the SEALN module for joint timbre/emotion control, Speaker Consistency Loss, and exploration of formant shift and speaker anonymization perturbations, plus the benefit of combining explicit prosodic features with pretrained latent features. Comprehensive evaluations demonstrate that EMM-TTS achieves superior naturalness, emotion transferability, and timbre consistency across languages, with favorable trade-offs between expressive control and speaker identity preservation.

Abstract

Cross-lingual emotional text-to-speech (TTS) aims to produce speech in one language that captures the emotion of a speaker from another language while maintaining the target voice's timbre. This process of cross-lingual emotional speech synthesis presents a complex challenge, necessitating flexible control over emotion, timbre, and language. However, emotion and timbre are highly entangled in speech signals, making fine-grained control challenging. To address this issue, we propose EMM-TTS, a novel two-stage cross-lingual emotional speech synthesis framework based on perturbed self-supervised learning (SSL) representations. In the first stage, the model explicitly and implicitly encodes prosodic cues to capture emotional expressiveness, while the second stage restores the timbre from perturbed SSL representations. We further investigate the effect of different speaker perturbation strategies-formant shifting and speaker anonymization-on the disentanglement of emotion and timbre. To strengthen speaker preservation and expressive control, we introduce Speaker Consistency Loss (SCL) and Speaker-Emotion Adaptive Layer Normalization (SEALN) modules. Additionally, we find that incorporating explicit acoustic features (e.g., F0, energy, and duration) alongside pretrained latent features improves voice cloning performance. Comprehensive multi-metric evaluations, including both subjective and objective measures, demonstrate that EMM-TTS achieves superior naturalness, emotion transferability, and timbre consistency across languages.

Perturbation Self-Supervised Representations for Cross-Lingual Emotion TTS: Stage-Wise Modeling of Emotion and Speaker

TL;DR

The paper tackles cross-lingual emotion TTS, where emotion from a speaker in one language is rendered in another language while preserving timbre. It introduces EMM-TTS, a two-stage framework that leverages perturbed self-supervised representations to decouple emotion from timbre, with stage 1 predicting emotion representations and stage 2 restoring timbre. Key contributions include the SEALN module for joint timbre/emotion control, Speaker Consistency Loss, and exploration of formant shift and speaker anonymization perturbations, plus the benefit of combining explicit prosodic features with pretrained latent features. Comprehensive evaluations demonstrate that EMM-TTS achieves superior naturalness, emotion transferability, and timbre consistency across languages, with favorable trade-offs between expressive control and speaker identity preservation.

Abstract

Cross-lingual emotional text-to-speech (TTS) aims to produce speech in one language that captures the emotion of a speaker from another language while maintaining the target voice's timbre. This process of cross-lingual emotional speech synthesis presents a complex challenge, necessitating flexible control over emotion, timbre, and language. However, emotion and timbre are highly entangled in speech signals, making fine-grained control challenging. To address this issue, we propose EMM-TTS, a novel two-stage cross-lingual emotional speech synthesis framework based on perturbed self-supervised learning (SSL) representations. In the first stage, the model explicitly and implicitly encodes prosodic cues to capture emotional expressiveness, while the second stage restores the timbre from perturbed SSL representations. We further investigate the effect of different speaker perturbation strategies-formant shifting and speaker anonymization-on the disentanglement of emotion and timbre. To strengthen speaker preservation and expressive control, we introduce Speaker Consistency Loss (SCL) and Speaker-Emotion Adaptive Layer Normalization (SEALN) modules. Additionally, we find that incorporating explicit acoustic features (e.g., F0, energy, and duration) alongside pretrained latent features improves voice cloning performance. Comprehensive multi-metric evaluations, including both subjective and objective measures, demonstrate that EMM-TTS achieves superior naturalness, emotion transferability, and timbre consistency across languages.

Paper Structure

This paper contains 24 sections, 3 equations, 4 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Propose method
  3. Problem definition and model review
  4. Emotion-dependent representations prediction
  5. Speech generation via speaker-perturbation representations
  6. EXPERIMENTS
  7. Data and Preprocessing
  8. Model and Training Setup
  9. Monolingual Voice Coling
  10. Cross-lingual Emotion Synthesis
  11. Evaluation Metrics
  12. Objective evaluation
  13. Subjective evaluation
  14. Experiment Results
  15. Performance on monolingual voice cloning
  16. ...and 9 more sections

Figures (4)

  • Figure 1: The problem definition of cross-lingual emotion speech synthesis.
  • Figure 2: Overview of the proposed EMM-TTS systems. The top figure presents an overview of the entire framework. The lower-left part illustrates the emotion-dependent representation prediction module, while the lower-right part shows the speech generation module based on speaker-perturbation representations.
  • Figure 3: Visualization of speaker embeddings under different speaker perturbation conditions. Different colors representing different speak ers.
  • Figure 4: ABX test results for speech synthesized by the EMM-TTS model using two different speaker perturbation strategies.