Self-Supervised Singing Voice Pre-Training towards Speech-to-Singing Conversion
Ruiqi Li, Rongjie Huang, Yongqi Wang, Zhiqing Hong, Zhou Zhao
TL;DR
The paper tackles data scarcity and alignment challenges in speech-to-singing conversion by introducing SVPT, a self-supervised Singing Voice Pre-Training framework built on a multi-scale decoder-only Transformer. It leverages discrete-unit perturbations (pitch/timbre and rhythm) and expanded-range reference prompting to learn from unannotated singing data and enable zero-shot conversion, while providing a pathway to high-quality SVS through an auxiliary text-to-semantic translator. Empirical results show SVPT outperforms baselines on STS and supports zero-shot SVS demonstrations, highlighting the value of cross-lingual pre-training and in-context timbre conditioning. The approach offers practical implications for scalable STS/SVS systems, albeit with notable computational demands and considerations around copyright and practical deployment.
Abstract
Speech-to-singing voice conversion (STS) task always suffers from data scarcity, because it requires paired speech and singing data. Compounding this issue are the challenges of content-pitch alignment and the suboptimal quality of generated outputs, presenting significant hurdles in STS research. This paper presents SVPT, an STS approach boosted by a self-supervised singing voice pre-training model. We leverage spoken language model techniques to tackle the rhythm alignment problem and the in-context learning capability to achieve zero-shot conversion. We adopt discrete-unit random resampling and pitch corruption strategies, enabling training with unpaired singing data and thus mitigating the issue of data scarcity. SVPT also serves as an effective backbone for singing voice synthesis (SVS), offering insights into scaling up SVS models. Experimental results indicate that SVPT delivers notable improvements in both STS and SVS endeavors. Audio samples are available at https://speech2sing.github.io.