Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Zero Shot Audio to Audio Emotion Transfer With Speaker Disentanglement

Soumya Dutta, Sriram Ganapathy

TL;DR

This work tackles audio-to-audio emotion style transfer under zero-shot conditions, aiming to transfer emotion from a target signal while preserving content and speaker identity from a source. It introduces ZEST, a disentangled, non-parallel framework that factorizes speech into HuBERT-based content tokens, emotion-agnostic speaker embeddings, and emotion embeddings, with a cross-attention pitch predictor and a HiFi-GAN-based reconstructor. The approach enables zero-shot transfers to unseen emotions and speakers and demonstrates improved emotion transfer accuracy while maintaining speech quality and speaker preservation, validated through both objective metrics and subjective listening tests. The work advances practical EST by leveraging self-supervised representations and adversarial disentanglement to perform emotion transfer without transcriptions or parallel data, with potential impact on expressive speech synthesis and human-computer interaction.

Abstract

The problem of audio-to-audio (A2A) style transfer involves replacing the style features of the source audio with those from the target audio while preserving the content related attributes of the source audio. In this paper, we propose an efficient approach, termed as Zero-shot Emotion Style Transfer (ZEST), that allows the transfer of emotional content present in the given source audio with the one embedded in the target audio while retaining the speaker and speech content from the source. The proposed system builds upon decomposing speech into semantic tokens, speaker representations and emotion embeddings. Using these factors, we propose a framework to reconstruct the pitch contour of the given speech signal and train a decoder that reconstructs the speech signal. The model is trained using a self-supervision based reconstruction loss. During conversion, the emotion embedding is alone derived from the target audio, while rest of the factors are derived from the source audio. In our experiments, we show that, even without using parallel training data or labels from the source or target audio, we illustrate zero shot emotion transfer capabilities of the proposed ZEST model using objective and subjective quality evaluations.

Zero Shot Audio to Audio Emotion Transfer With Speaker Disentanglement

TL;DR

This work tackles audio-to-audio emotion style transfer under zero-shot conditions, aiming to transfer emotion from a target signal while preserving content and speaker identity from a source. It introduces ZEST, a disentangled, non-parallel framework that factorizes speech into HuBERT-based content tokens, emotion-agnostic speaker embeddings, and emotion embeddings, with a cross-attention pitch predictor and a HiFi-GAN-based reconstructor. The approach enables zero-shot transfers to unseen emotions and speakers and demonstrates improved emotion transfer accuracy while maintaining speech quality and speaker preservation, validated through both objective metrics and subjective listening tests. The work advances practical EST by leveraging self-supervised representations and adversarial disentanglement to perform emotion transfer without transcriptions or parallel data, with potential impact on expressive speech synthesis and human-computer interaction.

Abstract

The problem of audio-to-audio (A2A) style transfer involves replacing the style features of the source audio with those from the target audio while preserving the content related attributes of the source audio. In this paper, we propose an efficient approach, termed as Zero-shot Emotion Style Transfer (ZEST), that allows the transfer of emotional content present in the given source audio with the one embedded in the target audio while retaining the speaker and speech content from the source. The proposed system builds upon decomposing speech into semantic tokens, speaker representations and emotion embeddings. Using these factors, we propose a framework to reconstruct the pitch contour of the given speech signal and train a decoder that reconstructs the speech signal. The model is trained using a self-supervision based reconstruction loss. During conversion, the emotion embedding is alone derived from the target audio, while rest of the factors are derived from the source audio. In our experiments, we show that, even without using parallel training data or labels from the source or target audio, we illustrate zero shot emotion transfer capabilities of the proposed ZEST model using objective and subjective quality evaluations.
Paper Structure (16 sections, 1 equation, 3 figures, 1 table)

This paper contains 16 sections, 1 equation, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Prior Works
  3. Method
  4. Content encoder
  5. Emotion Agnostic Speaker Encoder (EASE)
  6. Speaker Adversarial Classifier of Emotions (SACE)
  7. Pitch Contour Predictor
  8. Speech reconstruction
  9. Emotion conversion
  10. Experiments and Results
  11. Datasets and Pretraining
  12. Implementation
  13. Evaluation settings
  14. Objective evaluation
  15. Subjective tests
  16. ...and 1 more sections

Figures (3)

  • Figure 1: Training of the F0 contour predictor. EASE - Emotion Agnostic Speaker Encoder. SACE - Speaker Adversarial Classifier of Emotions. The blue blocks are kept frozen during training.
  • Figure 2: (a) During training, ZEST is learned to reconstruct the speech signal. (b) During emotion conversion, the components that are derived from target speech are coded in orange color. The yellow blocks are the learnable parts of the model using an auto-encoding objective, while the blue blocks indicate frozen components.
  • Figure 3: Subjective evaluation on the different test settings. Abbreviations used: MOS- Mean Opinion Score, SMOS - Similarity Mean Opinion Score. The definition of the different test settings is given in Sec. \ref{['sec:objective-evaluation']}. indicates that the difference in scores from the SSST test setting is statistically significant (p $<$ 0.05)