Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SelfTTS: cross-speaker style transfer through explicit embedding disentanglement and self-refinement using self-augmentation

Lucas H. Ueda, João G. T. Lima, Pedro R. Corrêa, Flávio O. Simões, Mário U. Neto, Paula D. P. Costa

Abstract

This paper presents SelfTTS, a text-to-speech (TTS) model designed for cross-speaker style transfer that eliminates the need for external pre-trained speaker or emotion encoders. The architecture achieves emotional expressivity in neutral speakers through an explicit disentanglement strategy utilizing Gradient Reversal Layers (GRL) combined with cosine similarity loss to decouple speaker and emotion information. We introduce Multi Positive Contrastive Learning (MPCL) to induce clustered representations of speaker and emotion embeddings based on their respective labels. Furthermore, SelfTTS employs a self-refinement strategy via Self-Augmentation, exploiting the model's voice conversion capabilities to enhance the naturalness of synthesized speech. Experimental results demonstrate that SelfTTS achieves superior emotional naturalness (eMOS) and robust stability in target timbre and emotion compared to state-of-the-art baselines.

SelfTTS: cross-speaker style transfer through explicit embedding disentanglement and self-refinement using self-augmentation

Abstract

This paper presents SelfTTS, a text-to-speech (TTS) model designed for cross-speaker style transfer that eliminates the need for external pre-trained speaker or emotion encoders. The architecture achieves emotional expressivity in neutral speakers through an explicit disentanglement strategy utilizing Gradient Reversal Layers (GRL) combined with cosine similarity loss to decouple speaker and emotion information. We introduce Multi Positive Contrastive Learning (MPCL) to induce clustered representations of speaker and emotion embeddings based on their respective labels. Furthermore, SelfTTS employs a self-refinement strategy via Self-Augmentation, exploiting the model's voice conversion capabilities to enhance the naturalness of synthesized speech. Experimental results demonstrate that SelfTTS achieves superior emotional naturalness (eMOS) and robust stability in target timbre and emotion compared to state-of-the-art baselines.
Paper Structure (19 sections, 10 equations, 3 figures, 7 tables)

This paper contains 19 sections, 10 equations, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. SelfTTS
  4. Multi Positive Contrastive Learning
  5. Explicit Embedding Disentanglement
  6. Self-Augmentation
  7. Experimental Setup
  8. Data
  9. Evaluation
  10. Results
  11. Towards Developing SelfTTS
  12. Loss-based Inductive Bias
  13. Does Emotion Encoder Input and Architecture Matter?
  14. Self-Augmentation
  15. Cross-corpus
  16. ...and 4 more sections

Figures (3)

  • Figure 1: SelfTTS model architecture. The emotion and speaker encoders receive mel-spectrogram slices of the reference waveform as input. Each encoder is optimized using the MPCL loss, while their embeddings are disentangled through a cosine-based GRL applied on top of the Linear Processor output for each encoder. The final forward step of the Normalizing Flows ($z_p$) is also disentangled using a cosine-based GRL, through a Convolutional Processor that predicts the corresponding emotion or speaker embedding. SDP stands for Stochastic Duration Predictor. Purple dashed arrows indicate the proposed Self-Augmentation pipeline.
  • Figure 2: Emotional similarity (eMOS) per emotion category across different models. The models are ordered as: GT (ground-truth), SelfTTS, SelfTTS w/o Self-Aug., VECL, and E3-VITS.
  • Figure 3: UMAP projections of the emotion style spaces for all evaluated models. Embeddings are colored by emotion: Neutral (blue), Happy (green), Angry (red), Sad (pink), and Surprise (orange). The emotion style space of SelfTTS clearly forms well-separated emotional clusters. VECL generates highly concentrated clusters, but there is still overlap between different emotions. E3-VITS is not capable of producing clustered representations.