Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EmoFake: An Initial Dataset for Emotion Fake Audio Detection

Yan Zhao, Jiangyan Yi, Jianhua Tao, Chenglong Wang, Xiaohui Zhang, Yongfeng Dong

TL;DR

This work introduces EmoFake, the first dataset specifically targeting emotion fake audio created by emotion voice conversion, built on the emotional speech database (ESD) with bilingual, multi-speaker data across five emotions. It evaluates detection and emotion-recognition baselines, showing that emotion tampering degrades existing fake-audio detectors trained on standard datasets, and proposes that graph-attention based detection with deep emotion embeddings (GADE) can better cope with such attacks. The study analyzes the impact of seven open-source EVC models on emotion transfer quality and detector robustness, revealing language- and model-dependent variations and underscoring the need for emotion-aware defenses. Baselines retrained with EmoFake data generally improve detection performance, indicating EmoFake’s utility for programmatic hardening of systems against emotionally manipulated speech. The work concludes with plans to expand EmoFake to more languages and emission intensities and to incorporate additional EVC models.

Abstract

Many datasets have been designed to further the development of fake audio detection, such as datasets of the ASVspoof and ADD challenges. However, these datasets do not consider a situation that the emotion of the audio has been changed from one to another, while other information (e.g. speaker identity and content) remains the same. Changing the emotion of an audio can lead to semantic changes. Speech with tampered semantics may pose threats to people's lives. Therefore, this paper reports our progress in developing such an emotion fake audio detection dataset involving changing emotion state of the origin audio named EmoFake. The fake audio in EmoFake is generated by open source emotion voice conversion models. Furthermore, we proposed a method named Graph Attention networks using Deep Emotion embedding (GADE) for the detection of emotion fake audio. Some benchmark experiments are conducted on this dataset. The results show that our designed dataset poses a challenge to the fake audio detection model trained with the LA dataset of ASVspoof 2019. The proposed GADE shows good performance in the face of emotion fake audio.

EmoFake: An Initial Dataset for Emotion Fake Audio Detection

TL;DR

This work introduces EmoFake, the first dataset specifically targeting emotion fake audio created by emotion voice conversion, built on the emotional speech database (ESD) with bilingual, multi-speaker data across five emotions. It evaluates detection and emotion-recognition baselines, showing that emotion tampering degrades existing fake-audio detectors trained on standard datasets, and proposes that graph-attention based detection with deep emotion embeddings (GADE) can better cope with such attacks. The study analyzes the impact of seven open-source EVC models on emotion transfer quality and detector robustness, revealing language- and model-dependent variations and underscoring the need for emotion-aware defenses. Baselines retrained with EmoFake data generally improve detection performance, indicating EmoFake’s utility for programmatic hardening of systems against emotionally manipulated speech. The work concludes with plans to expand EmoFake to more languages and emission intensities and to incorporate additional EVC models.

Abstract

Many datasets have been designed to further the development of fake audio detection, such as datasets of the ASVspoof and ADD challenges. However, these datasets do not consider a situation that the emotion of the audio has been changed from one to another, while other information (e.g. speaker identity and content) remains the same. Changing the emotion of an audio can lead to semantic changes. Speech with tampered semantics may pose threats to people's lives. Therefore, this paper reports our progress in developing such an emotion fake audio detection dataset involving changing emotion state of the origin audio named EmoFake. The fake audio in EmoFake is generated by open source emotion voice conversion models. Furthermore, we proposed a method named Graph Attention networks using Deep Emotion embedding (GADE) for the detection of emotion fake audio. Some benchmark experiments are conducted on this dataset. The results show that our designed dataset poses a challenge to the fake audio detection model trained with the LA dataset of ASVspoof 2019. The proposed GADE shows good performance in the face of emotion fake audio.
Paper Structure (11 sections, 6 equations, 2 figures, 9 tables)

This paper contains 11 sections, 6 equations, 2 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Dataset design
  3. Fake audio generation
  4. Dataset composition
  5. EXPERIMENTS
  6. Experiments setup
  7. Evaluation metrics
  8. Performance of speech emotion recognition
  9. Performance of fake audio detection models
  10. Baselines trained with EmoFake dataset
  11. CONCLUSION

Figures (2)

  • Figure 1: The emoiton fake audio changed the source emotion like surprise into the target emotion like angry without changing identity of speaker, content or other information.
  • Figure 2: The confusion matrix of emotion recognition experiments.(a) is the confusion matrix tested on the English subset of ESD. (b) is the confusion matrix tested on the English subset of EmoFake. (c) is the confusion matrix tested on the Chinese subset of ESD. (d) is the confusion matrix tested on the Chinese subset of EmoFake.