Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VoiceGuider: Enhancing Out-of-Domain Performance in Parameter-Efficient Speaker-Adaptive Text-to-Speech via Autoguidance

Jiheum Yeom, Heeseung Kim, Jooyoung Choi, Che Hyun Lee, Nohil Park, Sungroh Yoon

TL;DR

This work targets the OoD performance drop observed when using LoRA-based parameter-efficient finetuning for speaker-adaptive TTS. It introduces VoiceGuider, a diffusion-based, parameter-efficient one-shot TTS method that leverages autoguidance to mitigate LoRA errors by combining a strong model with an inferior one and by exploring inferior-model strategies and guidance intervals. The approach is validated on LibriTTS, VCTK, and especially GigaSpeech, showing that VoiceGuider can match full-finetuning performance on extreme OoD data while preserving naturalness and achieving superior speaker similarity over the LoRA baseline. The results suggest practical, scalable personalization for real-world, in-the-wild speech scenarios with significantly reduced parameter overhead.

Abstract

When applying parameter-efficient finetuning via LoRA onto speaker adaptive text-to-speech models, adaptation performance may decline compared to full-finetuned counterparts, especially for out-of-domain speakers. Here, we propose VoiceGuider, a parameter-efficient speaker adaptive text-to-speech system reinforced with autoguidance to enhance the speaker adaptation performance, reducing the gap against full-finetuned models. We carefully explore various ways of strengthening autoguidance, ultimately finding the optimal strategy. VoiceGuider as a result shows robust adaptation performance especially on extreme out-of-domain speech data. We provide audible samples in our demo page.

VoiceGuider: Enhancing Out-of-Domain Performance in Parameter-Efficient Speaker-Adaptive Text-to-Speech via Autoguidance

TL;DR

This work targets the OoD performance drop observed when using LoRA-based parameter-efficient finetuning for speaker-adaptive TTS. It introduces VoiceGuider, a diffusion-based, parameter-efficient one-shot TTS method that leverages autoguidance to mitigate LoRA errors by combining a strong model with an inferior one and by exploring inferior-model strategies and guidance intervals. The approach is validated on LibriTTS, VCTK, and especially GigaSpeech, showing that VoiceGuider can match full-finetuning performance on extreme OoD data while preserving naturalness and achieving superior speaker similarity over the LoRA baseline. The results suggest practical, scalable personalization for real-world, in-the-wild speech scenarios with significantly reduced parameter overhead.

Abstract

When applying parameter-efficient finetuning via LoRA onto speaker adaptive text-to-speech models, adaptation performance may decline compared to full-finetuned counterparts, especially for out-of-domain speakers. Here, we propose VoiceGuider, a parameter-efficient speaker adaptive text-to-speech system reinforced with autoguidance to enhance the speaker adaptation performance, reducing the gap against full-finetuned models. We carefully explore various ways of strengthening autoguidance, ultimately finding the optimal strategy. VoiceGuider as a result shows robust adaptation performance especially on extreme out-of-domain speech data. We provide audible samples in our demo page.
Paper Structure (14 sections, 4 equations, 1 figure, 4 tables)

This paper contains 14 sections, 4 equations, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Background
  4. VoiceTailor: Parameter-Efficient Baseline
  5. VoiceGuider: Eliminating LoRA Error with Autoguidance
  6. Experiments and Results
  7. Experimental Setup
  8. Datasets
  9. Training details
  10. Evaluation settings
  11. Problem Statement Verification
  12. Model Comparison
  13. Ablation Studies
  14. Conclusion

Figures (1)

  • Figure 1: Overview of VoiceGuider. VoiceGuider is a parameter-efficient one-shot TTS method for out-of-domain speakers. To eliminate errors caused by the parameter-efficient adapter, VoiceGuider enhances the prediction of the adapted model $s_1$ by pushing away from the predictions of the inferior model $s_0$. We explore two key ingredients of VoiceGuider: the method for obtaining the inferior model (left) and the guidance interval (right).