Analytic Study of Text-Free Speech Synthesis for Raw Audio using a Self-Supervised Learning Model
Joonyong Park, Daisuke Saito, Nobuaki Minematsu
TL;DR
This paper investigates text-free speech synthesis by comparing self-supervised learning (SSL) representations of raw audio against traditional text input, using a GSLM-like pipeline with speech2unit, unit2speech, and a uLM. It evaluates three input representations—ground-truth text, ASR-derived text, and SSL discrete symbols—across English and Japanese, focusing on intelligibility, naturalness, and acoustic quality. Key findings show that text inputs preserve semantic content best, while SSL discrete symbols better capture acoustic and paralinguistic cues, with SSL performance strongly affected by language dependency and codebook size. The work highlights the potential for zero-resource data augmentation in multilingual speech synthesis and points to multilingual tokenization as a path to reducing language bias.
Abstract
We examine the text-free speech representations of raw audio obtained from a self-supervised learning (SSL) model by analyzing the synthesized speech using the SSL representations instead of conventional text representations. Since raw audio does not have paired speech representations as transcribed texts do, obtaining speech representations from unpaired speech is crucial for augmenting available datasets for speech synthesis. Specifically, the proposed speech synthesis is conducted using discrete symbol representations from the SSL model in comparison with text representations, and analytical examinations of the synthesized speech have been carried out. The results empirically show that using text representations is advantageous for preserving semantic information, while using discrete symbol representations is superior for preserving acoustic content, including prosodic and intonational information.