VoiceTailor: Lightweight Plug-In Adapter for Diffusion-Based Personalized Text-to-Speech
Heeseung Kim, Sang-gil Lee, Jiheum Yeom, Che Hyun Lee, Sungwon Kim, Sungroh Yoon
TL;DR
VoiceTailor addresses the high cost of adapting TTS to new speakers by proposing a diffusion-based, parameter-efficient approach that injects LoRA adapters into attention modules identified via weight-change analysis. Built on UnitSpeech with a unit encoder, it uses classifier-free guidance and targeted guidance strategies to strengthen speaker information during sampling. The method achieves speaker adaptation performance comparable to full fine-tuning while using only 0.25% of parameters (about 311K of 127M, ~1.3 MB), and demonstrates robustness across real-world speakers with short adaptation data. This work significantly lowers the data and compute burden of personalized TTS, enabling scalable, real-world deployment for many speakers.
Abstract
We propose VoiceTailor, a parameter-efficient speaker-adaptive text-to-speech (TTS) system, by equipping a pre-trained diffusion-based TTS model with a personalized adapter. VoiceTailor identifies pivotal modules that benefit from the adapter based on a weight change ratio analysis. We utilize Low-Rank Adaptation (LoRA) as a parameter-efficient adaptation method and incorporate the adapter into pivotal modules of the pre-trained diffusion decoder. To achieve powerful adaptation performance with few parameters, we explore various guidance techniques for speaker adaptation and investigate the best strategies to strengthen speaker information. VoiceTailor demonstrates comparable speaker adaptation performance to existing adaptive TTS models by fine-tuning only 0.25\% of the total parameters. VoiceTailor shows strong robustness when adapting to a wide range of real-world speakers, as shown in the demo.