Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Everyone-Can-Sing: Zero-Shot Singing Voice Synthesis and Conversion with Speech Reference

Shuqi Dai, Yunyun Wang, Roger B. Dannenberg, Zeyu Jin

TL;DR

The paper tackles zero-shot cross-domain SVS and SVC by proposing a unified framework that disentangles voice timbre, lyrics content, and expressive performance, and leverages abundant speech data through mixed training. It introduces three models (one SVS and two SVC variants) built on a diffusion-based synthesizer and pre-trained embeddings (Resemblyzer for timbre, GR0 for content) to enable precise control over language, performance attributes, singing style, and voice identity using only a short speech reference. The approach demonstrates improved timbre similarity and musicality over state-of-the-art baselines in subjective evaluations and provides ablations that illuminate the impact of mixed training, pitch adaptation, and content embedding strategies. This work advances practical cross-domain singing generation and suggests broader applicability to low-data music tasks, including instrumental timbre style transfer.

Abstract

We propose a unified framework for Singing Voice Synthesis (SVS) and Conversion (SVC), addressing the limitations of existing approaches in cross-domain SVS/SVC, poor output musicality, and scarcity of singing data. Our framework enables control over multiple aspects, including language content based on lyrics, performance attributes based on a musical score, singing style and vocal techniques based on a selector, and voice identity based on a speech sample. The proposed zero-shot learning paradigm consists of one SVS model and two SVC models, utilizing pre-trained content embeddings and a diffusion-based generator. The proposed framework is also trained on mixed datasets comprising both singing and speech audio, allowing singing voice cloning based on speech reference. Experiments show substantial improvements in timbre similarity and musicality over state-of-the-art baselines, providing insights into other low-data music tasks such as instrumental style transfer. Examples can be found at: everyone-can-sing.github.io.

Everyone-Can-Sing: Zero-Shot Singing Voice Synthesis and Conversion with Speech Reference

TL;DR

The paper tackles zero-shot cross-domain SVS and SVC by proposing a unified framework that disentangles voice timbre, lyrics content, and expressive performance, and leverages abundant speech data through mixed training. It introduces three models (one SVS and two SVC variants) built on a diffusion-based synthesizer and pre-trained embeddings (Resemblyzer for timbre, GR0 for content) to enable precise control over language, performance attributes, singing style, and voice identity using only a short speech reference. The approach demonstrates improved timbre similarity and musicality over state-of-the-art baselines in subjective evaluations and provides ablations that illuminate the impact of mixed training, pitch adaptation, and content embedding strategies. This work advances practical cross-domain singing generation and suggests broader applicability to low-data music tasks, including instrumental timbre style transfer.

Abstract

We propose a unified framework for Singing Voice Synthesis (SVS) and Conversion (SVC), addressing the limitations of existing approaches in cross-domain SVS/SVC, poor output musicality, and scarcity of singing data. Our framework enables control over multiple aspects, including language content based on lyrics, performance attributes based on a musical score, singing style and vocal techniques based on a selector, and voice identity based on a speech sample. The proposed zero-shot learning paradigm consists of one SVS model and two SVC models, utilizing pre-trained content embeddings and a diffusion-based generator. The proposed framework is also trained on mixed datasets comprising both singing and speech audio, allowing singing voice cloning based on speech reference. Experiments show substantial improvements in timbre similarity and musicality over state-of-the-art baselines, providing insights into other low-data music tasks such as instrumental style transfer. Examples can be found at: everyone-can-sing.github.io.
Paper Structure (12 sections, 1 figure, 2 tables)

This paper contains 12 sections, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Unified Framework
  4. Zero-Shot Singing Voice Synthesis
  5. Zero-Shot SVC Given Lyrics Alignment
  6. Zero-Shot SVC With Local Content Embedding
  7. Experiments
  8. Experiment Settings
  9. Subjective Evaluation
  10. Comparison Study
  11. Ablation Study
  12. Conclusion

Figures (1)

  • Figure 1: Proposed zero-shot SVS and SVC framework.