Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The VoicePrivacy 2024 Challenge Evaluation Plan

Natalia Tomashenko, Xiaoxiao Miao, Pierre Champion, Sarina Meyer, Xin Wang, Emmanuel Vincent, Michele Panariello, Nicholas Evans, Junichi Yamagishi, Massimiliano Todisco

TL;DR

This paper presents the VoicePrivacy 2024 Challenge Evaluation Plan, outlining an objective framework for evaluating utterance-level voice anonymization that conceals speaker identity while preserving linguistic content and emotional state. The approach combines a rigorous privacy-utility tradeoff framework with a strong attacker model, fixed downstream task models, and a suite of public baselines (B1–B6) to benchmark progress. It provides a comprehensive data ecosystem, training resources, and a detailed, multilayer evaluation protocol using privacy (EER) and utility (WER, UAR) metrics across multiple operating points, all within an accessible, Python-centered workflow. The plan aims to standardize comparisons, accelerate progress in practical privacy-preserving speech technologies, and enable reproducible assessments in real-world GDPR-like contexts.

Abstract

The task of the challenge is to develop a voice anonymization system for speech data which conceals the speaker's voice identity while protecting linguistic content and emotional states. The organizers provide development and evaluation datasets and evaluation scripts, as well as baseline anonymization systems and a list of training resources formed on the basis of the participants' requests. Participants apply their developed anonymization systems, run evaluation scripts and submit evaluation results and anonymized speech data to the organizers. Results will be presented at a workshop held in conjunction with Interspeech 2024 to which all participants are invited to present their challenge systems and to submit additional workshop papers.

The VoicePrivacy 2024 Challenge Evaluation Plan

TL;DR

This paper presents the VoicePrivacy 2024 Challenge Evaluation Plan, outlining an objective framework for evaluating utterance-level voice anonymization that conceals speaker identity while preserving linguistic content and emotional state. The approach combines a rigorous privacy-utility tradeoff framework with a strong attacker model, fixed downstream task models, and a suite of public baselines (B1–B6) to benchmark progress. It provides a comprehensive data ecosystem, training resources, and a detailed, multilayer evaluation protocol using privacy (EER) and utility (WER, UAR) metrics across multiple operating points, all within an accessible, Python-centered workflow. The plan aims to standardize comparisons, accelerate progress in practical privacy-preserving speech technologies, and enable reproducible assessments in real-world GDPR-like contexts.

Abstract

The task of the challenge is to develop a voice anonymization system for speech data which conceals the speaker's voice identity while protecting linguistic content and emotional states. The organizers provide development and evaluation datasets and evaluation scripts, as well as baseline anonymization systems and a list of training resources formed on the basis of the participants' requests. Participants apply their developed anonymization systems, run evaluation scripts and submit evaluation results and anonymized speech data to the organizers. Results will be presented at a workshop held in conjunction with Interspeech 2024 to which all participants are invited to present their challenge systems and to submit additional workshop papers.
Paper Structure (28 sections, 2 equations, 7 figures, 11 tables)

This paper contains 28 sections, 2 equations, 7 figures, 11 tables.

Table of Contents

  1. Challenge objectives
  2. Task
  3. Voice anonymization task
  4. Attack model
  5. Data and pretrained models
  6. Training resources
  7. Development and evaluation data
  8. Privacy and utility evaluation
  9. Objective assessment of the privacy-utility tradeoff
  10. Privacy metric: equal error rate (EER)
  11. Utility metrics
  12. Word error rate (WER)
  13. Unweighted average recall (UAR)
  14. Baseline voice anonymization systems
  15. Anonymization using x-vectors and a neural source-filter model: B1
  16. ...and 13 more sections

Figures (7)

  • Figure 1: Example system rankings according to the privacy (EER) and utility (WER and UAR) results for 4 minimum target EERs. Different colors correspond to 6 different teams. Numbers within each circle show system ranks for a given condition. Grey circles correspond to the baseline systems, and the black circle to the original (unprocessed) data.
  • Figure 2: Privacy and utility evaluation.
  • Figure 3: Baseline anonymization system B1.
  • Figure 4: Baseline anonymization system B2.
  • Figure 5: Baseline anonymization system B3.
  • ...and 2 more figures