An Empirical Study of Speech Language Models for Prompt-Conditioned Speech Synthesis

TL;DR

The paper investigates how prompt design and content semantic units affect speech synthesis in autoregressive and non-autoregressive speech LMs. Through an empirical evaluation using HuBERT-based content units and EnCodec acoustic units on a large English corpus, the study reveals that heterogeneous and nonstationary prompts degrade style transfer, and that content carries rich acoustic information that leaks into synthesized speech. Prosody—pitch, tempo, volume, and emphasis—cannot be fully controlled by prompts, with pitch aligned to prompts in AR and tempo largely dictated by content. The findings highlight the need for more disentangled speech representations and improved modeling to achieve robust, controllable prompt-conditioned speech synthesis, and the authors provide code for replication.

Abstract

Speech language models (LMs) are promising for high-quality speech synthesis through in-context learning. A typical speech LM takes discrete semantic units as content and a short utterance as prompt, and synthesizes speech which preserves the content's semantics but mimics the prompt's style. However, there is no systematic understanding on how the synthesized audio is controlled by the prompt and content. In this work, we conduct an empirical study of the widely used autoregressive (AR) and non-autoregressive (NAR) speech LMs and provide insights into the prompt design and content semantic units. Our analysis reveals that heterogeneous and nonstationary prompts hurt the audio quality in contrast to the previous finding that longer prompts always lead to better synthesis. Moreover, we find that the speaker style of the synthesized audio is also affected by the content in addition to the prompt. We further show that semantic units carry rich acoustic information such as pitch, tempo, volume and speech emphasis, which might be leaked from the content to the synthesized audio.

Figures (1)

• Figure 1: Overview of the primary task of speech LMs and inference procedures of AR and NAR LMs.