Neural Multi-Speaker Voice Cloning for Nepali in Low-Resource Settings

TL;DR

The paper addresses few-shot, multi-speaker voice cloning for Nepali in low-resource settings by constructing two data streams and adapting a Tacotron2-based TTS with a speaker encoder and WaveRNN vocoder. It demonstrates that speaker embeddings learned from untranscribed Nepali audio can be fused with text representations to clone unseen voices with reasonable naturalness. Key contributions include a large Nepali speaker-encoder dataset, a curated paired Nepali dataset for synthesis, and quantitative evaluations (cosine similarity, EER, MOS) showing robust cloning performance. The work advances accessible, personalized speech synthesis for Nepali and provides a foundation for further development with larger datasets and advanced vocoders.

Abstract

This research presents a few-shot voice cloning system for Nepali speakers, designed to synthesize speech in a specific speaker's voice from Devanagari text using minimal data. Voice cloning in Nepali remains largely unexplored due to its low-resource nature. To address this, we constructed separate datasets: untranscribed audio for training a speaker encoder and paired text-audio data for training a Tacotron2-based synthesizer. The speaker encoder, optimized with Generative End2End loss, generates embeddings that capture the speaker's vocal identity, validated through Uniform Manifold Approximation and Projection (UMAP) for dimension reduction visualizations. These embeddings are fused with Tacotron2's text embeddings to produce mel-spectrograms, which are then converted into audio using a WaveRNN vocoder. Audio data were collected from various sources, including self-recordings, and underwent thorough preprocessing for quality and alignment. Training was performed using mel and gate loss functions under multiple hyperparameter settings. The system effectively clones speaker characteristics even for unseen voices, demonstrating the feasibility of few-shot voice cloning for the Nepali language and establishing a foundation for personalized speech synthesis in low-resource scenarios.

Figures (30)

• Figure 1: Age distribution of male and female speakers across different age groups for our self-collected dataset for speaker encoder
• Figure 2: Average number of utterances per speaker.
• Figure 3: Audio duration of the collected dataset and the number of audio files for Synthesizer data
• Figure 4: Architecture of the Nepali voice cloning system combining a Speaker Encoder, Tacotron2-based TTS synthesizer, and WaveRNN vocoder.
• Figure 5: Equal Error Rate (EER) curve during training of the speaker encoder. The steady decline indicates improved discrimination of speaker embeddings over time.