LDM-SVC: Latent Diffusion Model Based Zero-Shot Any-to-Any Singing Voice Conversion with Singer Guidance
Shihao Chen, Yu Gu, Jie Zhang, Na Li, Rilin Chen, Liping Chen, Lirong Dai
TL;DR
The paper tackles any-to-any singing voice conversion (SVC) with the timbre leakage problem. It introduces LDM-SVC, which operates in a latent space derived from a pre-trained So-VITS-SVC VAE and employs a latent diffusion model conditioned on PPG, F0, and singer embeddings, plus a singer-guided, classifier-free training scheme to suppress the source timbre. The approach yields higher timbre similarity and naturalness than several baselines in both seen and unseen singer scenarios, with particularly notable gains in zero-shot conversions due to the singer guidance. This work advances zero-shot SVC by combining latent-space diffusion with trainable timbre suppression, potentially enabling more robust cross-domain and low-resource singing voice editing.
Abstract
Any-to-any singing voice conversion (SVC) is an interesting audio editing technique, aiming to convert the singing voice of one singer into that of another, given only a few seconds of singing data. However, during the conversion process, the issue of timbre leakage is inevitable: the converted singing voice still sounds like the original singer's voice. To tackle this, we propose a latent diffusion model for SVC (LDM-SVC) in this work, which attempts to perform SVC in the latent space using an LDM. We pretrain a variational autoencoder structure using the noted open-source So-VITS-SVC project based on the VITS framework, which is then used for the LDM training. Besides, we propose a singer guidance training method based on classifier-free guidance to further suppress the timbre of the original singer. Experimental results show the superiority of the proposed method over previous works in both subjective and objective evaluations of timbre similarity.