NanoVoice: Efficient Speaker-Adaptive Text-to-Speech for Multiple Speakers

TL;DR

NanoVoice tackles efficient speaker adaptation for multi-speaker TTS by batch-adapting multiple references in parallel using LoRA-based, rank-$2$ adapters. It combines a batch-wise fine-tuning scheme with selective parameter sharing and introduces a lightweight trainable scale matrix to preserve performance, achieving comparable quality to one-shot baselines while being ~4x faster and using ~45% fewer adaptation parameters with 40 references. The approach leverages VoiceTailor as a backbone, extends it with a shared $B$ matrix and $A'$ adapters, and employs a normalization-based scale to mitigate sharing drawbacks. Overall, NanoVoice enables scalable, cost-effective personalized TTS suitable for commercial deployment and supports robust multi-speaker adaptation with limited data per speaker.

Abstract

We present NanoVoice, a personalized text-to-speech model that efficiently constructs voice adapters for multiple speakers simultaneously. NanoVoice introduces a batch-wise speaker adaptation technique capable of fine-tuning multiple references in parallel, significantly reducing training time. Beyond building separate adapters for each speaker, we also propose a parameter sharing technique that reduces the number of parameters used for speaker adaptation. By incorporating a novel trainable scale matrix, NanoVoice mitigates potential performance degradation during parameter sharing. NanoVoice achieves performance comparable to the baselines, while training 4 times faster and using 45 percent fewer parameters for speaker adaptation with 40 reference voices. Extensive ablation studies and analysis further validate the efficiency of our model.

Figures (1)

• Figure 1: An overview of speaker adaptation in various scenarios: single reference adaptation, sequential adaptation of multiple references, and NanoVoice.