Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adversarial Multi-Task Learning for Disentangling Timbre and Pitch in Singing Voice Synthesis

Tae-Woo Kim, Min-Su Kang, Gyeong-Hoon Lee

TL;DR

This work tackles singing voice synthesis by addressing timbre-pitch entanglement through adversarial multi-task learning. It introduces a dual-encoder, dual-decoder SVS model guided by auxiliary predictions of WORLD-vocoder features (MGC, BAP, V/UV, logF0) to disentangle timbre and pitch, and a mel-predictor that combines these cues for final mel-spectrogram generation. The model is trained in two phases and enhanced by three discriminators within a GAN framework to boost perceptual quality, with evaluations showing improvements over single-task baselines and conventional WORLD-based methods, especially when using a Parallel WaveGAN vocoder. The approach demonstrates stronger naturalness in multi-singer SVS and provides a route to more controllable vocal timbre and pitch in neural singing synthesis. The proposed framework offers practical impact for high-fidelity, expressive SVS systems and informs future improvements in vocoder performance at high sampling rates.

Abstract

Recently, deep learning-based generative models have been introduced to generate singing voices. One approach is to predict the parametric vocoder features consisting of explicit speech parameters. This approach has the advantage that the meaning of each feature is explicitly distinguished. Another approach is to predict mel-spectrograms for a neural vocoder. However, parametric vocoders have limitations of voice quality and the mel-spectrogram features are difficult to model because the timbre and pitch information are entangled. In this study, we propose a singing voice synthesis model with multi-task learning to use both approaches -- acoustic features for a parametric vocoder and mel-spectrograms for a neural vocoder. By using the parametric vocoder features as auxiliary features, the proposed model can efficiently disentangle and control the timbre and pitch components of the mel-spectrogram. Moreover, a generative adversarial network framework is applied to improve the quality of singing voices in a multi-singer model. Experimental results demonstrate that our proposed model can generate more natural singing voices than the single-task models, while performing better than the conventional parametric vocoder-based model.

Adversarial Multi-Task Learning for Disentangling Timbre and Pitch in Singing Voice Synthesis

TL;DR

This work tackles singing voice synthesis by addressing timbre-pitch entanglement through adversarial multi-task learning. It introduces a dual-encoder, dual-decoder SVS model guided by auxiliary predictions of WORLD-vocoder features (MGC, BAP, V/UV, logF0) to disentangle timbre and pitch, and a mel-predictor that combines these cues for final mel-spectrogram generation. The model is trained in two phases and enhanced by three discriminators within a GAN framework to boost perceptual quality, with evaluations showing improvements over single-task baselines and conventional WORLD-based methods, especially when using a Parallel WaveGAN vocoder. The approach demonstrates stronger naturalness in multi-singer SVS and provides a route to more controllable vocal timbre and pitch in neural singing synthesis. The proposed framework offers practical impact for high-fidelity, expressive SVS systems and informs future improvements in vocoder performance at high sampling rates.

Abstract

Recently, deep learning-based generative models have been introduced to generate singing voices. One approach is to predict the parametric vocoder features consisting of explicit speech parameters. This approach has the advantage that the meaning of each feature is explicitly distinguished. Another approach is to predict mel-spectrograms for a neural vocoder. However, parametric vocoders have limitations of voice quality and the mel-spectrogram features are difficult to model because the timbre and pitch information are entangled. In this study, we propose a singing voice synthesis model with multi-task learning to use both approaches -- acoustic features for a parametric vocoder and mel-spectrograms for a neural vocoder. By using the parametric vocoder features as auxiliary features, the proposed model can efficiently disentangle and control the timbre and pitch components of the mel-spectrogram. Moreover, a generative adversarial network framework is applied to improve the quality of singing voices in a multi-singer model. Experimental results demonstrate that our proposed model can generate more natural singing voices than the single-task models, while performing better than the conventional parametric vocoder-based model.
Paper Structure (16 sections, 5 equations, 3 figures, 1 table)

This paper contains 16 sections, 5 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related works
  3. Proposed methods
  4. Embeddings
  5. Timbre and pitch decoders
  6. Mel-predictor
  7. Discriminators
  8. Experiments
  9. Experimental setups
  10. Dataset
  11. Model configurations
  12. Training
  13. Results
  14. Listening test
  15. Effect of adversarial training for auxiliary task
  16. ...and 1 more sections

Figures (3)

  • Figure 1: Architecture overview. (a) is a single-task SVS model, (b) is the proposed multi-task SVS model, (c) is a mel-predictor, and (d) is a discriminator.
  • Figure 2: Generated mel-spectrograms of (a) final predicted output, (b) timbre representation, and (c) pitch representation. The top-line figures represent spectrograms from the model trained using adversarial loss for mel-spectrogram alone, while the bottom is from the model using adversarial loss for all features.
  • Figure 3: A/B preference test for expressiveness of the generated samples from the proposed model with or without adversarial training for auxiliary task.