Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Robust Training of Singing Voice Synthesis Using Prior and Posterior Uncertainty

Yiwen Zhao, Jiatong Shi, Yuxun Tang, William Chen, Shinji Watanabe

TL;DR

This paper tackles data scarcity in singing voice synthesis by introducing uncertainty-based training: differentiable augmentation to enlarge prior uncertainty and a frame-level posterior uncertainty predictor to focus learning on uncertain regions. Built on the VISinger2 framework, the methods are designed to preserve target distributions while expanding variability, improving performance on long-tail singing patterns. Empirical results across Mandarin Opencpop and Japanese Ofuton-P show consistent gains in pitch accuracy, timbre, duration, and perceptual naturalness, with the combined approach achieving the strongest improvements. The work demonstrates robust, data-efficient SVS training and broadens applicability to cross-language singing generation.

Abstract

Singing voice synthesis (SVS) has seen remarkable advancements in recent years. However, compared to speech and general audio data, publicly available singing datasets remain limited. In practice, this data scarcity often leads to performance degradation in long-tail scenarios, such as imbalanced pitch distributions or rare singing styles. To mitigate these challenges, we propose uncertainty-based optimization to improve the training process of end-to-end SVS models. First, we introduce differentiable data augmentation in the adversarial training, which operates in a sample-wise manner to increase the prior uncertainty. Second, we incorporate a frame-level uncertainty prediction module that estimates the posterior uncertainty, enabling the model to allocate more learning capacity to low-confidence segments. Empirical results on the Opencpop and Ofuton-P, across Chinese and Japanese, demonstrate that our approach improves performance in various perspectives.

Robust Training of Singing Voice Synthesis Using Prior and Posterior Uncertainty

TL;DR

This paper tackles data scarcity in singing voice synthesis by introducing uncertainty-based training: differentiable augmentation to enlarge prior uncertainty and a frame-level posterior uncertainty predictor to focus learning on uncertain regions. Built on the VISinger2 framework, the methods are designed to preserve target distributions while expanding variability, improving performance on long-tail singing patterns. Empirical results across Mandarin Opencpop and Japanese Ofuton-P show consistent gains in pitch accuracy, timbre, duration, and perceptual naturalness, with the combined approach achieving the strongest improvements. The work demonstrates robust, data-efficient SVS training and broadens applicability to cross-language singing generation.

Abstract

Singing voice synthesis (SVS) has seen remarkable advancements in recent years. However, compared to speech and general audio data, publicly available singing datasets remain limited. In practice, this data scarcity often leads to performance degradation in long-tail scenarios, such as imbalanced pitch distributions or rare singing styles. To mitigate these challenges, we propose uncertainty-based optimization to improve the training process of end-to-end SVS models. First, we introduce differentiable data augmentation in the adversarial training, which operates in a sample-wise manner to increase the prior uncertainty. Second, we incorporate a frame-level uncertainty prediction module that estimates the posterior uncertainty, enabling the model to allocate more learning capacity to low-confidence segments. Empirical results on the Opencpop and Ofuton-P, across Chinese and Japanese, demonstrate that our approach improves performance in various perspectives.

Paper Structure

This paper contains 20 sections, 16 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Singing Voice Synthesis
  4. Data Augmentation
  5. Uncertainty Prediction
  6. Formulation
  7. VISinger2 Framework
  8. Differentiable Augmentation
  9. Uncertainty Prediction
  10. Experiment
  11. Training Strategies
  12. Corpora
  13. Evaluation Metrics
  14. Quantitative Analysis
  15. Qualitative Analysis
  16. ...and 5 more sections

Figures (2)

  • Figure 1: The pipeline of VISinger2 with our proposed uncertainty predictor and differentiable augmentation. The uncertainty predictor is trained with a forward generator pass, and the differentiable augmentation is added to spectrogram-level features in adversarial training.
  • Figure 2: Spectrogram case study. The green box shows the system using the uncertainty predictor, and the differentiable augmentation produces a clearer phoneme boundary, which indicates faithful lyrics.