Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Why disentanglement-based speaker anonymization systems fail at preserving emotions?

Ünal Ege Gaznepoglu, Nils Peters

TL;DR

This paper addresses the problem of preserving emotion information in disentanglement-based speaker anonymization. It conducts a comprehensive ablation of the SSL-SAS system, varying the intermediate representation ($ ext{IR}$), speaker embeddings, and vocoder behavior to identify factors that degrade emotion signals, and evaluates with metrics including $EER$, $UAR$, $WER$, and $SER$ on LibriTest and IEMOCAP. The main finding is that insufficient emotion-related information in the $IR$ is the primary cause of emotion loss, with speaker embeddings learned in a generative context also containing emotion cues, while vocoder out-of-distribution performance has a smaller effect. The work suggests directions to distill emotion information from self-supervised representations and to sanitize embeddings, guiding the design of more emotion-preserving anonymization approaches in practical settings.

Abstract

Disentanglement-based speaker anonymization involves decomposing speech into a semantically meaningful representation, altering the speaker embedding, and resynthesizing a waveform using a neural vocoder. State-of-the-art systems of this kind are known to remove emotion information. Possible reasons include mode collapse in GAN-based vocoders, unintended modeling and modification of emotions through speaker embeddings, or excessive sanitization of the intermediate representation. In this paper, we conduct a comprehensive evaluation of a state-of-the-art speaker anonymization system to understand the underlying causes. We conclude that the main reason is the lack of emotion-related information in the intermediate representation. The speaker embeddings also have a high impact, if they are learned in a generative context. The vocoder's out-of-distribution performance has a smaller impact. Additionally, we discovered that synthesis artifacts increase spectral kurtosis, biasing emotion recognition evaluation towards classifying utterances as angry. Therefore, we conclude that reporting unweighted average recall alone for emotion recognition performance is suboptimal.

Why disentanglement-based speaker anonymization systems fail at preserving emotions?

TL;DR

This paper addresses the problem of preserving emotion information in disentanglement-based speaker anonymization. It conducts a comprehensive ablation of the SSL-SAS system, varying the intermediate representation (), speaker embeddings, and vocoder behavior to identify factors that degrade emotion signals, and evaluates with metrics including , , , and on LibriTest and IEMOCAP. The main finding is that insufficient emotion-related information in the is the primary cause of emotion loss, with speaker embeddings learned in a generative context also containing emotion cues, while vocoder out-of-distribution performance has a smaller effect. The work suggests directions to distill emotion information from self-supervised representations and to sanitize embeddings, guiding the design of more emotion-preserving anonymization approaches in practical settings.

Abstract

Disentanglement-based speaker anonymization involves decomposing speech into a semantically meaningful representation, altering the speaker embedding, and resynthesizing a waveform using a neural vocoder. State-of-the-art systems of this kind are known to remove emotion information. Possible reasons include mode collapse in GAN-based vocoders, unintended modeling and modification of emotions through speaker embeddings, or excessive sanitization of the intermediate representation. In this paper, we conduct a comprehensive evaluation of a state-of-the-art speaker anonymization system to understand the underlying causes. We conclude that the main reason is the lack of emotion-related information in the intermediate representation. The speaker embeddings also have a high impact, if they are learned in a generative context. The vocoder's out-of-distribution performance has a smaller impact. Additionally, we discovered that synthesis artifacts increase spectral kurtosis, biasing emotion recognition evaluation towards classifying utterances as angry. Therefore, we conclude that reporting unweighted average recall alone for emotion recognition performance is suboptimal.
Paper Structure (19 sections, 4 figures, 2 tables)

This paper contains 19 sections, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Hypotheses
  3. Mode collapse of the GAN-based vocoders
  4. Speaker embeddings capturing emotion data
  5. Discretization of the intermediate representation
  6. Methodology
  7. System Under Test
  8. The experiments
  9. Evaluation Metrics
  10. Results
  11. Discretization of the intermediate representation
  12. Speaker embeddings capturing emotion data
  13. Mode collapse of the GAN-based vocoders
  14. What happens to emotion-related acoustic features?
  15. Spectral Centroid
  16. ...and 4 more sections

Figures (4)

  • Figure 1: The signal flow diagram of the SSL-SAS system.
  • Figure 2: Privacy evaluation for speaker anonymization. Top: Unprotected case, for reference. Bottom: The lazy-informed attack model, comparing the resulting speech for two separate invocations of the anonymization system.
  • Figure 3: Utility evaluation for speaker anonymization. Left: Emotion recognition, Right: Intelligibility via ASR.
  • Figure 4: Distributions of emotion-related acoustic features per experiment. Spectral centroid and spectral kurtosis are plotted in log-scale. Top: on IEMOCAP-test, Bottom: on Libri-test