PolySinger: Singing-Voice to Singing-Voice Translation from English to Japanese

TL;DR

This paper introduces PolySinger, the first modular system for singing-voice to singing-voice translation (SV2SVT) that translates English lyrics into Japanese and synthesizes a corresponding Japanese singing performance. The pipeline cascades automatic lyrics transcription, phoneme- and syllable-level alignment, frame-level melody extraction, English→Japanese lyrics translation with pronunciation decoding, and SVS using Synthesizer V. A MOS-based evaluation with native Japanese listeners shows a solid foundational framework for SV2SVT but reveals that naturalness and faithful translation remain areas for improvement, with no significant gains from fine-tuning over a baseline model. The work highlights practical challenges in linguistic prosody, G2P for Japanese, and dataset limitations, and outlines future directions such as sentiment-aware lyrics generation and autonomous natural Japanese lyric creation to advance SV2SVT toward higher-quality, cross-lingual singing synthesis.

Abstract

The speech domain prevails in the spotlight for several natural language processing (NLP) tasks while the singing domain remains less explored. The culmination of NLP is the speech-to-speech translation (S2ST) task, referring to translation and synthesis of human speech. A disparity between S2ST and the possible adaptation to the singing domain, which we describe as singing-voice to singing-voice translation (SV2SVT), is becoming prominent as the former is progressing ever faster, while the latter is at a standstill. Singing-voice synthesis systems are overcoming the barrier of multi-lingual synthesis, despite limited attention has been paid to multi-lingual songwriting and song translation. This paper endeavors to determine what is required for successful SV2SVT and proposes PolySinger (Polyglot Singer): the first system for SV2SVT, performing lyrics translation from English to Japanese. A cascaded approach is proposed to establish a framework with a high degree of control which can potentially diminish the disparity between SV2SVT and S2ST. The performance of PolySinger is evaluated by a mean opinion score test with native Japanese speakers. Results and in-depth discussions with test subjects suggest a solid foundation for SV2SVT, but several shortcomings must be overcome, which are discussed for the future of SV2SVT.

Figures (3)

• Figure 1: Overview of our proposed SV2SVT system, PolySinger. Provided an English vocal performance, a synthetic vocal performance is created in Block 7, defined by notes with onsets, durations and Japanese lyrics, guided by a frame-level melody. Every numeric value is in seconds and "< >" illustrates the boundaries of notes. The process of segmenting words into syllables is illustrated in Table \ref{['tab:syllables']}. Fundamentals of Japanese writing are explained in Subsection \ref{['subsec:translation']}, and the process of converting kanji to hiragana is illustrated in Table \ref{['tab:kanji']}.
• Figure 2: Bar plots representing per-question score's relative frequency from the MOS quality test for Baseline (left) and Fine-tuned (right).
• Figure :