Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generative Data Augmentation Challenge: Zero-Shot Speech Synthesis for Personalized Speech Enhancement

Jae-Sung Bae, Anastasia Kuznetsova, Dinesh Manocha, John Hershey, Trausti Kristjansson, Minje Kim

TL;DR

The paper introduces a Generative Data Augmentation Challenge at ICASSP 2025 that uses zero-shot speech synthesis to augment data for personalized speech enhancement (PSE). It investigates how the quality of synthetic, target-speaker data affects downstream PSE performance, and provides baseline experiments with open-source zero-shot TTS models to benchmark progress. The study leverages real LibriTTS speakers and Meta-generated virtual speakers, plus MUSAN noise, and evaluates both TTS (speaker similarity, intelligibility, perceptual quality) and PSE (SDRI, SDR, eSTOI, PESQ) metrics. Findings indicate that higher-quality augmented data improves PSE outcomes, with data quality and speaker resemblance playing crucial roles, and also highlight privacy-friendly avenues via virtual speakers. The work lays groundwork for broader use of generative data augmentation in personalized speech systems and invites extension to more downstream tasks.

Abstract

This paper presents a new challenge that calls for zero-shot text-to-speech (TTS) systems to augment speech data for the downstream task, personalized speech enhancement (PSE), as part of the Generative Data Augmentation workshop at ICASSP 2025. Collecting high-quality personalized data is challenging due to privacy concerns and technical difficulties in recording audio from the test scene. To address these issues, synthetic data generation using generative models has gained significant attention. In this challenge, participants are tasked first with building zero-shot TTS systems to augment personalized data. Subsequently, PSE systems are asked to be trained with this augmented personalized dataset. Through this challenge, we aim to investigate how the quality of augmented data generated by zero-shot TTS models affects PSE model performance. We also provide baseline experiments using open-source zero-shot TTS models to encourage participation and benchmark advancements. Our baseline code implementation and checkpoints are available online.

Generative Data Augmentation Challenge: Zero-Shot Speech Synthesis for Personalized Speech Enhancement

TL;DR

The paper introduces a Generative Data Augmentation Challenge at ICASSP 2025 that uses zero-shot speech synthesis to augment data for personalized speech enhancement (PSE). It investigates how the quality of synthetic, target-speaker data affects downstream PSE performance, and provides baseline experiments with open-source zero-shot TTS models to benchmark progress. The study leverages real LibriTTS speakers and Meta-generated virtual speakers, plus MUSAN noise, and evaluates both TTS (speaker similarity, intelligibility, perceptual quality) and PSE (SDRI, SDR, eSTOI, PESQ) metrics. Findings indicate that higher-quality augmented data improves PSE outcomes, with data quality and speaker resemblance playing crucial roles, and also highlight privacy-friendly avenues via virtual speakers. The work lays groundwork for broader use of generative data augmentation in personalized speech systems and invites extension to more downstream tasks.

Abstract

This paper presents a new challenge that calls for zero-shot text-to-speech (TTS) systems to augment speech data for the downstream task, personalized speech enhancement (PSE), as part of the Generative Data Augmentation workshop at ICASSP 2025. Collecting high-quality personalized data is challenging due to privacy concerns and technical difficulties in recording audio from the test scene. To address these issues, synthetic data generation using generative models has gained significant attention. In this challenge, participants are tasked first with building zero-shot TTS systems to augment personalized data. Subsequently, PSE systems are asked to be trained with this augmented personalized dataset. Through this challenge, we aim to investigate how the quality of augmented data generated by zero-shot TTS models affects PSE model performance. We also provide baseline experiments using open-source zero-shot TTS models to encourage participation and benchmark advancements. Our baseline code implementation and checkpoints are available online.
Paper Structure (16 sections, 1 figure, 5 tables)

This paper contains 16 sections, 1 figure, 5 tables.

Table of Contents

  1. Introduction
  2. Challenge Description
  3. Zero-Shot TTS Models
  4. The Downstream Personalized Speech Enhancement Task
  5. Challenge Datasets
  6. Real-World Speakers
  7. Virtual Speakers
  8. Noise Sources
  9. Evaluation Metrics
  10. Basic TTS Performance Evaluation
  11. Evaluation metrics for PSE models
  12. Baseline Models
  13. Baseline zero-shot TTS models
  14. Baseline PSE models
  15. Results
  16. ...and 1 more sections

Figures (1)

  • Figure 1: Overall flow of our challenge. (a) First, participants augment the personalized data using a zero-shot TTS model. (b) Next, organizers ask participants to fine-tune the PSE model with this augmented personalized data, followed by an inference phase in which enhanced speech is generated for evaluation.