Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

When Fine-Tuning Fails and when it Generalises: Role of Data Diversity and Mixed Training in LLM-based TTS

Anupam Purwar, Aditya Choudhary

TL;DR

Overall this work establishes that LoRA finetuning is not merely a parameter efficient optimization technique but an effective mechanism for better speaker level adaptation in compact LLM-based TTS systems.

Abstract

Large language models are increasingly adopted as semantic backbones for neural text-to-speech systems. However, frozen LLM representations are insufficient for modeling speaker specific acoustic and perceptual characteristics. Our experiments involving fine tuning of the Language Model backbone of TTS show promise in improving the voice consistency and Signal to Noise ratio SNR in voice cloning task. Across multiple speakers LoRA finetuning consistently outperforms the non-finetuned base Qwen-0.5B model across three complementary dimensions of speech quality. First, perceptual quality improves significantly with DNS-MOS gains of up to 0.42 points for speakers whose training data exhibits sufficient acoustic variability. Second, speaker fidelity improves for all evaluated speakers with consistent increases in voice similarity indicating that LoRA effectively adapts speaker identity representations without degrading linguistic modeling. Third, signal level quality improves in most cases with signal to noise ratio increasing by as much as 34 percent. Crucially these improvements are strongly governed by the characteristics of the training data. Speakers with high variability in acoustic energy and perceptual quality achieve simultaneous gains in DNS-MOS voice similarity and SNR. Overall this work establishes that LoRA finetuning is not merely a parameter efficient optimization technique but an effective mechanism for better speaker level adaptation in compact LLM-based TTS systems. When supported by sufficiently diverse training data LoRA adapted Qwen-0.5B consistently surpasses its frozen base model in perceptual quality speaker similarity with low latency using GGUF model hosted in quantized form.

When Fine-Tuning Fails and when it Generalises: Role of Data Diversity and Mixed Training in LLM-based TTS

TL;DR

Overall this work establishes that LoRA finetuning is not merely a parameter efficient optimization technique but an effective mechanism for better speaker level adaptation in compact LLM-based TTS systems.

Abstract

Large language models are increasingly adopted as semantic backbones for neural text-to-speech systems. However, frozen LLM representations are insufficient for modeling speaker specific acoustic and perceptual characteristics. Our experiments involving fine tuning of the Language Model backbone of TTS show promise in improving the voice consistency and Signal to Noise ratio SNR in voice cloning task. Across multiple speakers LoRA finetuning consistently outperforms the non-finetuned base Qwen-0.5B model across three complementary dimensions of speech quality. First, perceptual quality improves significantly with DNS-MOS gains of up to 0.42 points for speakers whose training data exhibits sufficient acoustic variability. Second, speaker fidelity improves for all evaluated speakers with consistent increases in voice similarity indicating that LoRA effectively adapts speaker identity representations without degrading linguistic modeling. Third, signal level quality improves in most cases with signal to noise ratio increasing by as much as 34 percent. Crucially these improvements are strongly governed by the characteristics of the training data. Speakers with high variability in acoustic energy and perceptual quality achieve simultaneous gains in DNS-MOS voice similarity and SNR. Overall this work establishes that LoRA finetuning is not merely a parameter efficient optimization technique but an effective mechanism for better speaker level adaptation in compact LLM-based TTS systems. When supported by sufficiently diverse training data LoRA adapted Qwen-0.5B consistently surpasses its frozen base model in perceptual quality speaker similarity with low latency using GGUF model hosted in quantized form.
Paper Structure (21 sections, 3 figures, 14 tables)

This paper contains 21 sections, 3 figures, 14 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Full Finetuning
  4. LoRA Finetuning
  5. Datasets
  6. Metrics
  7. Results
  8. DNS-MOS Quantitative Comparison
  9. Perceptual Trends and Analysis
  10. Discussions
  11. Metrics: Mean Opinion Score (MOS)
  12. Discussion: Finetuning epochs and effect on MOS
  13. LoRA Fine-tuning : Impact on Inference latency
  14. Impact of Training Data: Energy Variability, DNS-MOS Dispersion
  15. Effect of context audio length
  16. ...and 6 more sections

Figures (3)

  • Figure 1: End-to-End Pipeline of a cascading voice agent comprising of a ASR module to transcribe voice to text, LLM module to reason over the text and generate text output and lastly a TTS module to synthesize speech for the reasoned text. Our work to the pipeline focuses on finetuning the LLM backbone (enclosed in red) responsible for token prediction, which are subsequently decoded by the neural codec to generate the waveform
  • Figure 2: Plots of Training and Validation Loss at various training steps for each speaker ID. The curves indicate stochastic decrease of training loss with training steps with saturation in validation loss as we approach 4 epochs
  • Figure 3: Plots indicating MOS Score (Y axis) of Audio Generated with varying lengths of cloning audio provided (X axis). Base refers to the standard NeuTTS model, Ref refers to the audio used for cloning. "Long" refers to generation of a longer sentence (180 characters) whereas other audios have 70 characters.