Scale This, Not That: Investigating Key Dataset Attributes for Efficient Speech Enhancement Scaling
Leying Zhang, Wangyou Zhang, Chenda Li, Yanmin Qian
TL;DR
The paper tackles the challenge of understanding how individual dataset attributes impact speech enhancement (SE) performance as models scale. It introduces a generation-training-evaluation framework that uses a zero-shot TTS system to synthesize datasets with controlled variations in text, language, speaker, and noise, enabling attribute-wise analysis for both discriminative and generative SE models. Key findings show that semantic attributes (text, language) have limited impact, while acoustic attributes (speaker diversity and noise diversity) significantly influence generalization, with discriminative models benefiting more from noise variety. The work provides practical guidance for efficient data curation in SE scaling and demonstrates the viability of purely synthetic data for SE training, paving the way for broader, scalable data-centric studies.
Abstract
Recent speech enhancement models have shown impressive performance gains by scaling up model complexity and training data. However, the impact of dataset variability (e.g. text, language, speaker, and noise) has been underexplored. Analyzing each attribute individually is often challenging, as multiple attributes are usually entangled in commonly used datasets, posing a significant obstacle in understanding the distinct contributions of each attribute to the model's performance. To address this challenge, we propose a generation-training-evaluation framework that leverages zero-shot text-to-speech systems to investigate the impact of controlled attribute variations on speech enhancement performance. It enables us to synthesize training datasets in a scalable manner while carefully altering each attribute. Based on the proposed framework, we analyze the scaling effects of various dataset attributes on the performance of both discriminative and generative SE models. Extensive experiments on multi-domain corpora imply that acoustic attributes (e.g., speaker and noise) are much more important to current speech enhancement models than semantic attributes (e.g., language and text), offering new insights for future research.