NANSY++: Unified Voice Synthesis with Neural Analysis and Synthesis

TL;DR

NANSY++ introduces a self-supervised, modular backbone that decomposes voice into pitch, linguistic content, timbre, and excitation, enabling unified synthesis across voice conversion, TTS, SVS, and voice designing. The backbone employs self-supervised F0 estimation, disentangled linguistic representations, and time-varying timbre embeddings, paired with a two-stage waveform synthesizer to deliver high-quality outputs with data efficiency. The paper demonstrates strong results in reconstruction, zero-shot tasks, and data-limited singing and TTS scenarios, highlighting fast training and controllability over voice attributes. By consolidating analysis features into a shared synthesizer, NANSY++ advances co-creative workflows and reduces the need for large annotated datasets, while offering explicit mechanisms for age, gender, and voice identity manipulation.

Abstract

Various applications of voice synthesis have been developed independently despite the fact that they generate "voice" as output in common. In addition, most of the voice synthesis models still require a large number of audio data paired with annotated labels (e.g., text transcription and music score) for training. To this end, we propose a unified framework of synthesizing and manipulating voice signals from analysis features, dubbed NANSY++. The backbone network of NANSY++ is trained in a self-supervised manner that does not require any annotations paired with audio. After training the backbone network, we efficiently tackle four voice applications - i.e. voice conversion, text-to-speech, singing voice synthesis, and voice designing - by partially modeling the analysis features required for each task. Extensive experiments show that the proposed framework offers competitive advantages such as controllability, data efficiency, and fast training convergence, while providing high quality synthesis. Audio samples: tinyurl.com/8tnsy3uc.

Figures (19)

• Figure 1: Overview of proposed NANSY++ backbone architecture. All modules in the backbone network is trained in an end-to-end manner within a single analysis and synthesis loop.
• Figure 2: Overview of exemplar applications integrated into the NANSY++ backbone architecture. Each application can be substituted into a problem of estimating analysis features from the backbone.
• Figure 3: Age shift evaluation.
• Figure 4: Gender shift evaluation.
• Figure 5: The qualitative results on $F_0$ estimation. Left column shows the $F_0$ estimation results from rapt algorithm. Right column shows the $F_0$ estimation results from NANSY++ pitch encoder. First row shows the $F_0$ estimation results from clean signal. Second row shows the $F_0$ estimation results from noisy signal. The results clearly demonstrates that the pitch encoder is indeed estimating $F_0$, even robustly for noisy signal.