Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RDSinger: Reference-based Diffusion Network for Singing Voice Synthesis

Kehan Sui, Jinxu Xiang, Fang Jin

TL;DR

RDSinger is introduced, a reference-based denoising diffusion network that generates high-quality audio for SVS tasks and outperforms current state-of-the-art SVS methods in performance.

Abstract

Singing voice synthesis (SVS) aims to produce high-fidelity singing audio from music scores, requiring a detailed understanding of notes, pitch, and duration, unlike text-to-speech tasks. Although diffusion models have shown exceptional performance in various generative tasks like image and video creation, their application in SVS is hindered by time complexity and the challenge of capturing acoustic features, particularly during pitch transitions. Some networks learn from the prior distribution and use the compressed latent state as a better start in the diffusion model, but the denoising step doesn't consistently improve quality over the entire duration. We introduce RDSinger, a reference-based denoising diffusion network that generates high-quality audio for SVS tasks. Our approach is inspired by Animate Anyone, a diffusion image network that maintains intricate appearance features from reference images. RDSinger utilizes FastSpeech2 mel-spectrogram as a reference to mitigate denoising step artifacts. Additionally, existing models could be influenced by misleading information on the compressed latent state during pitch transitions. We address this issue by applying Gaussian blur on partial reference mel-spectrogram and adjusting loss weights in these regions. Extensive ablation studies demonstrate the efficiency of our method. Evaluations on OpenCpop, a Chinese singing dataset, show that RDSinger outperforms current state-of-the-art SVS methods in performance.

RDSinger: Reference-based Diffusion Network for Singing Voice Synthesis

TL;DR

RDSinger is introduced, a reference-based denoising diffusion network that generates high-quality audio for SVS tasks and outperforms current state-of-the-art SVS methods in performance.

Abstract

Singing voice synthesis (SVS) aims to produce high-fidelity singing audio from music scores, requiring a detailed understanding of notes, pitch, and duration, unlike text-to-speech tasks. Although diffusion models have shown exceptional performance in various generative tasks like image and video creation, their application in SVS is hindered by time complexity and the challenge of capturing acoustic features, particularly during pitch transitions. Some networks learn from the prior distribution and use the compressed latent state as a better start in the diffusion model, but the denoising step doesn't consistently improve quality over the entire duration. We introduce RDSinger, a reference-based denoising diffusion network that generates high-quality audio for SVS tasks. Our approach is inspired by Animate Anyone, a diffusion image network that maintains intricate appearance features from reference images. RDSinger utilizes FastSpeech2 mel-spectrogram as a reference to mitigate denoising step artifacts. Additionally, existing models could be influenced by misleading information on the compressed latent state during pitch transitions. We address this issue by applying Gaussian blur on partial reference mel-spectrogram and adjusting loss weights in these regions. Extensive ablation studies demonstrate the efficiency of our method. Evaluations on OpenCpop, a Chinese singing dataset, show that RDSinger outperforms current state-of-the-art SVS methods in performance.

Paper Structure

This paper contains 23 sections, 12 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Neural Speech Processing
  4. Singing Voice synthesis
  5. Diffusion Probabilistic Model
  6. Methodology
  7. Diffusion Model
  8. Diffusion Process
  9. Reverse Process
  10. Training the Model
  11. Reference-based Diffusion
  12. Pitch Transition Enhancement
  13. Experiments
  14. Experimental Setup
  15. Dataset
  16. ...and 8 more sections

Figures (5)

  • Figure 1: The figure illustrates the pitch transition region. Panel (a) shows the pitch transition generated by FastSpeech2, with the orange line representing the high-to-low frequency ratio and white dots marking transition points. The red area highlights the transition zone. Panel (b) shows the alignment of FastSpeech2's pitch transition region with the ground-truth mel-spectrogram, indicating that the transition regions correspond to areas that need enhancement. Panel (c) displays the result of applying Gaussian blur to the pitch transition region shown in (a).
  • Figure 2: The training (left) and inference (right) process of RDSinger
  • Figure 3: The diffusion network structure of RDSinger
  • Figure 4: Visualization of generated samples with varying systems: (a) GT, (b) VISinger, (c) DiffSinger, (d)RDSinger.
  • Figure 5: Distribution of Ranking Scores for RDSinger