Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Preliminary Investigation on Flexible Singing Voice Synthesis Through Decomposed Framework with Inferrable Features

Lester Phillip Violeta, Taketo Akama

TL;DR

This paper tackles the data-label bottleneck in singing voice synthesis (SVS) by proposing SingFlex, a decomposed three-stage framework that infers linguistic content (HLFs), pitch (F0), and timbre features from audio. By substituting traditional labeled inputs with inferrable features and enabling language and singer adaptation, SingFlex reduces labeling requirements and introduces lyric inpainting, while leveraging diffusion-based waveform synthesis conditioned on HLFs, F0, and singer embeddings. The approach combines FastSpeech-based linguistic modeling, MIDI-guided pitch prediction from audio, and a HuBERT soft–based linguistic representation to support multilingual SVS and multi-singer synthesis. Experimental results show competitive intelligibility with a baseline SVS system, reveal the benefits of large-scale pretraining for reduced label dependence, and demonstrate successful language adaptation, singer transfer, and lyric inpainting, pointing to a flexible, multifunctional SVS system with practical impact in personalized singing experiences.

Abstract

We investigate the feasibility of a singing voice synthesis (SVS) system by using a decomposed framework to improve flexibility in generating singing voices. Due to data-driven approaches, SVS performs a music score-to-waveform mapping; however, the direct mapping limits control, such as being able to only synthesize in the language or the singers present in the labeled singing datasets. As collecting large singing datasets labeled with music scores is an expensive task, we investigate an alternative approach by decomposing the SVS system and inferring different singing voice features. We decompose the SVS system into three-stage modules of linguistic, pitch contour, and synthesis, in which singing voice features such as linguistic content, F0, voiced/unvoiced, singer embeddings, and loudness are directly inferred from audio. Through this decomposed framework, we show that we can alleviate the labeled dataset requirements, adapt to different languages or singers, and inpaint the lyrical content of singing voices. Our investigations show that the framework has the potential to reach state-of-the-art in SVS, even though the model has additional functionality and improved flexibility. The comprehensive analysis of our investigated framework's current capabilities sheds light on the ways the research community can achieve a flexible and multifunctional SVS system.

A Preliminary Investigation on Flexible Singing Voice Synthesis Through Decomposed Framework with Inferrable Features

TL;DR

This paper tackles the data-label bottleneck in singing voice synthesis (SVS) by proposing SingFlex, a decomposed three-stage framework that infers linguistic content (HLFs), pitch (F0), and timbre features from audio. By substituting traditional labeled inputs with inferrable features and enabling language and singer adaptation, SingFlex reduces labeling requirements and introduces lyric inpainting, while leveraging diffusion-based waveform synthesis conditioned on HLFs, F0, and singer embeddings. The approach combines FastSpeech-based linguistic modeling, MIDI-guided pitch prediction from audio, and a HuBERT soft–based linguistic representation to support multilingual SVS and multi-singer synthesis. Experimental results show competitive intelligibility with a baseline SVS system, reveal the benefits of large-scale pretraining for reduced label dependence, and demonstrate successful language adaptation, singer transfer, and lyric inpainting, pointing to a flexible, multifunctional SVS system with practical impact in personalized singing experiences.

Abstract

We investigate the feasibility of a singing voice synthesis (SVS) system by using a decomposed framework to improve flexibility in generating singing voices. Due to data-driven approaches, SVS performs a music score-to-waveform mapping; however, the direct mapping limits control, such as being able to only synthesize in the language or the singers present in the labeled singing datasets. As collecting large singing datasets labeled with music scores is an expensive task, we investigate an alternative approach by decomposing the SVS system and inferring different singing voice features. We decompose the SVS system into three-stage modules of linguistic, pitch contour, and synthesis, in which singing voice features such as linguistic content, F0, voiced/unvoiced, singer embeddings, and loudness are directly inferred from audio. Through this decomposed framework, we show that we can alleviate the labeled dataset requirements, adapt to different languages or singers, and inpaint the lyrical content of singing voices. Our investigations show that the framework has the potential to reach state-of-the-art in SVS, even though the model has additional functionality and improved flexibility. The comprehensive analysis of our investigated framework's current capabilities sheds light on the ways the research community can achieve a flexible and multifunctional SVS system.
Paper Structure (24 sections, 1 equation, 2 figures, 2 tables)

This paper contains 24 sections, 1 equation, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related literature
  3. Singing synthesis from music score inputs
  4. Singing synthesis from audio inputs
  5. Proposed Method
  6. Linguistic feature module: HLF generation from time-aligned text inputs
  7. Pitch contour module: F0 generation from MIDI inputs
  8. Singing synthesis module: Singing waveform generation from linguistic, pitch, and timbre features
  9. Inpainting procedure
  10. Experimental Setup
  11. Training and inference details
  12. Baseline comparison
  13. Datasets
  14. Evaluation methods
  15. Results and Discussion
  16. ...and 9 more sections

Figures (2)

  • Figure 1: An overview of the investigated method SingFlex. The text are transformed into HLFs, which are used to predict the pitch and other acoustic features. From these features, the singing waveforms are synthesized.
  • Figure 2: Detailed visualizations of each module and the datasets used to train each module.