Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

BnTTS: Few-Shot Speaker Adaptation in Low-Resource Setting

Mohammad Jahid Ibna Basher, Md Kowsher, Md Saiful Islam, Rabindra Nath Nandi, Nusrat Jahan Prottasha, Mehadi Hasan Menon, Tareq Al Muntasir, Shammur Absar Chowdhury, Firoj Alam, Niloofar Yousefi, Ozlem Ozmen Garibay

TL;DR

The paper tackles Bangla text-to-speech in low-resource settings by enabling few-shot speaker adaptation within a multilingual XTTS framework. It extends XTTS with Bangla-specific phonetic handling, a VQ-VAE-based audio representation, an LLM-driven synthesis head, and a HiFi-GAN vocoder to produce natural, speaker-consistent speech. Pretraining on approximately 3.85k hours of Bangla data and a small HQ speaker set enables effective few-shot adaptation, demonstrated on two evaluation corpora (BnStudioEval and BnTTSTextEval). Results show strong subjective gains for few-shot BnTTS-n over zero-shot and competitive performance against commercial baselines in several metrics, with remaining challenges in transcription-level metrics (CER) and short-text generation. The work releases public evaluation data and outlines ethical considerations and future directions toward broader dialect coverage and real-time inference.

Abstract

This paper introduces BnTTS (Bangla Text-To-Speech), the first framework for Bangla speaker adaptation-based TTS, designed to bridge the gap in Bangla speech synthesis using minimal training data. Building upon the XTTS architecture, our approach integrates Bangla into a multilingual TTS pipeline, with modifications to account for the phonetic and linguistic characteristics of the language. We pre-train BnTTS on 3.85k hours of Bangla speech dataset with corresponding text labels and evaluate performance in both zero-shot and few-shot settings on our proposed test dataset. Empirical evaluations in few-shot settings show that BnTTS significantly improves the naturalness, intelligibility, and speaker fidelity of synthesized Bangla speech. Compared to state-of-the-art Bangla TTS systems, BnTTS exhibits superior performance in Subjective Mean Opinion Score (SMOS), Naturalness, and Clarity metrics.

BnTTS: Few-Shot Speaker Adaptation in Low-Resource Setting

TL;DR

The paper tackles Bangla text-to-speech in low-resource settings by enabling few-shot speaker adaptation within a multilingual XTTS framework. It extends XTTS with Bangla-specific phonetic handling, a VQ-VAE-based audio representation, an LLM-driven synthesis head, and a HiFi-GAN vocoder to produce natural, speaker-consistent speech. Pretraining on approximately 3.85k hours of Bangla data and a small HQ speaker set enables effective few-shot adaptation, demonstrated on two evaluation corpora (BnStudioEval and BnTTSTextEval). Results show strong subjective gains for few-shot BnTTS-n over zero-shot and competitive performance against commercial baselines in several metrics, with remaining challenges in transcription-level metrics (CER) and short-text generation. The work releases public evaluation data and outlines ethical considerations and future directions toward broader dialect coverage and real-time inference.

Abstract

This paper introduces BnTTS (Bangla Text-To-Speech), the first framework for Bangla speaker adaptation-based TTS, designed to bridge the gap in Bangla speech synthesis using minimal training data. Building upon the XTTS architecture, our approach integrates Bangla into a multilingual TTS pipeline, with modifications to account for the phonetic and linguistic characteristics of the language. We pre-train BnTTS on 3.85k hours of Bangla speech dataset with corresponding text labels and evaluate performance in both zero-shot and few-shot settings on our proposed test dataset. Empirical evaluations in few-shot settings show that BnTTS significantly improves the naturalness, intelligibility, and speaker fidelity of synthesized Bangla speech. Compared to state-of-the-art Bangla TTS systems, BnTTS exhibits superior performance in Subjective Mean Opinion Score (SMOS), Naturalness, and Clarity metrics.

Paper Structure

This paper contains 26 sections, 14 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. BnTTS
  3. Experiments
  4. Results
  5. Related Works
  6. Conclusion
  7. Limitations
  8. Acknowledgments
  9. Ethical Considerations
  10. TTS Data Acquisition Framework
  11. Dataset Filtering Criteria
  12. Human Guided Data Preparation
  13. Dataset
  14. Evaluation Dataset
  15. Training Objectives
  16. ...and 11 more sections

Figures (3)

  • Figure 1: Overview of BnTTS Model.
  • Figure 2: Overview of our TTS Data Acquisition Framework. The acquisition process involves using a Speech-to-Text model to obtain transcription, an LLM to restore transcription's punctuation, a noise suppression model to remove unwanted noise, and finally an audio superresolution model to enhance audio quality and loudness.
  • Figure 3: These figures demonstrate how the ratio of text length to audio duration changes before and after processing the data.