Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Boosting Large Language Model for Speech Synthesis: An Empirical Study

Hongkun Hao, Long Zhou, Shujie Liu, Jinyu Li, Shujie Hu, Rui Wang, Furu Wei

TL;DR

The study investigates how to endow large language models with speech synthesis capabilities by comparing three integration strategies with VALL-E: direct fine-tuning of the LLM, a superposed LLM–VALL-E architecture, and a coupled LLM-as-text-encoder approach. It finds that LoRA-based direct fine-tuning underperforms the VALL-E baseline, while coupling LLMs and VALL-E yields the strongest improvements, including notable WER reductions and better speaker similarity and naturalness. The results highlight that codec codes are not simply another language and that leveraging LLMs as a text encoder to empower VALL-E is particularly effective. The work provides practical guidance on designing LLM-augmented TTS systems and underscores the value of pre-trained components and architectural coupling for high-quality speech generation.

Abstract

Large language models (LLMs) have made significant advancements in natural language processing and are concurrently extending the language ability to other modalities, such as speech and vision. Nevertheless, most of the previous work focuses on prompting LLMs with perception abilities like auditory comprehension, and the effective approach for augmenting LLMs with speech synthesis capabilities remains ambiguous. In this paper, we conduct a comprehensive empirical exploration of boosting LLMs with the ability to generate speech, by combining pre-trained LLM LLaMA/OPT and text-to-speech synthesis model VALL-E. We compare three integration methods between LLMs and speech synthesis models, including directly fine-tuned LLMs, superposed layers of LLMs and VALL-E, and coupled LLMs and VALL-E using LLMs as a powerful text encoder. Experimental results show that, using LoRA method to fine-tune LLMs directly to boost the speech synthesis capability does not work well, and superposed LLMs and VALL-E can improve the quality of generated speech both in speaker similarity and word error rate (WER). Among these three methods, coupled methods leveraging LLMs as the text encoder can achieve the best performance, making it outperform original speech synthesis models with a consistently better speaker similarity and a significant (10.9%) WER reduction.

Boosting Large Language Model for Speech Synthesis: An Empirical Study

TL;DR

The study investigates how to endow large language models with speech synthesis capabilities by comparing three integration strategies with VALL-E: direct fine-tuning of the LLM, a superposed LLM–VALL-E architecture, and a coupled LLM-as-text-encoder approach. It finds that LoRA-based direct fine-tuning underperforms the VALL-E baseline, while coupling LLMs and VALL-E yields the strongest improvements, including notable WER reductions and better speaker similarity and naturalness. The results highlight that codec codes are not simply another language and that leveraging LLMs as a text encoder to empower VALL-E is particularly effective. The work provides practical guidance on designing LLM-augmented TTS systems and underscores the value of pre-trained components and architectural coupling for high-quality speech generation.

Abstract

Large language models (LLMs) have made significant advancements in natural language processing and are concurrently extending the language ability to other modalities, such as speech and vision. Nevertheless, most of the previous work focuses on prompting LLMs with perception abilities like auditory comprehension, and the effective approach for augmenting LLMs with speech synthesis capabilities remains ambiguous. In this paper, we conduct a comprehensive empirical exploration of boosting LLMs with the ability to generate speech, by combining pre-trained LLM LLaMA/OPT and text-to-speech synthesis model VALL-E. We compare three integration methods between LLMs and speech synthesis models, including directly fine-tuned LLMs, superposed layers of LLMs and VALL-E, and coupled LLMs and VALL-E using LLMs as a powerful text encoder. Experimental results show that, using LoRA method to fine-tune LLMs directly to boost the speech synthesis capability does not work well, and superposed LLMs and VALL-E can improve the quality of generated speech both in speaker similarity and word error rate (WER). Among these three methods, coupled methods leveraging LLMs as the text encoder can achieve the best performance, making it outperform original speech synthesis models with a consistently better speaker similarity and a significant (10.9%) WER reduction.
Paper Structure (27 sections, 2 figures, 8 tables)

This paper contains 27 sections, 2 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Model Components
  5. Large Language Model
  6. Speech Compression Model
  7. Codec Language Model
  8. Integration Strategies
  9. Method A: Directly Fine-tuned LLMs
  10. Method B: Superposed LLMs and VALL-E
  11. Method C: Coupled LLMs and VALL-E
  12. Experiments
  13. Experiment Setup
  14. Dataset:
  15. Data Preprocessing:
  16. ...and 12 more sections

Figures (2)

  • Figure 1: Overview of the proposed different integration methods. (a) Method A: Directly fine-tuned LLMs where LLMs are trained for predicting codec codes with an expanded vocabulary. (b) Method B: Superposed LLMs and VALL-E, where both LLMs and VALL-E are used to model textual tokens and acoustic tokens successively. (c) Method C: Coupled LLMs and VALL-E, where the better text representation provided by LLM is regarded as the textual input of VALL-E.
  • Figure 2: WER results of using different model sizes in Method A under three inference strategies introduced in Section \ref{['sec:inference_strategies']}. The overall results including speaker similarity and speech naturalness are summarized in Appendix \ref{['sec:appendix_effect_model_size']}.