Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VoxHakka: A Dialectally Diverse Multi-speaker Text-to-Speech System for Taiwanese Hakka

Li-Wei Chen, Hung-Shin Lee, Chen-Chi Chang

TL;DR

VoxHakka addresses the data scarcity and pronunciation complexity of Taiwanese Hakka by delivering a zero-shot, dialect-aware TTS system built on YourTTS with dialect and speaker embeddings across six dialects. It collects and cleans a large public-data-derived corpus via ASR-based alignment, forced timing, and silence trimming, then trains a VAE-based end-to-end TTS pipeline with a WaveNet-augmented decoder and a HiFi-GAN vocoder. Objective and CMOS assessments show VoxHakka outperforms existing public Hakka TTS systems in naturalness and tone, with dialect-specific speaker embeddings providing the best naturalness. By releasing the model under CC-BY 4.0, the work offers a valuable resource for language preservation and future research in low-resource, multi-dialect speech technologies.

Abstract

This paper introduces VoxHakka, a text-to-speech (TTS) system designed for Taiwanese Hakka, a critically under-resourced language spoken in Taiwan. Leveraging the YourTTS framework, VoxHakka achieves high naturalness and accuracy and low real-time factor in speech synthesis while supporting six distinct Hakka dialects. This is achieved by training the model with dialect-specific data, allowing for the generation of speaker-aware Hakka speech. To address the scarcity of publicly available Hakka speech corpora, we employed a cost-effective approach utilizing a web scraping pipeline coupled with automatic speech recognition (ASR)-based data cleaning techniques. This process ensured the acquisition of a high-quality, multi-speaker, multi-dialect dataset suitable for TTS training. Subjective listening tests conducted using comparative mean opinion scores (CMOS) demonstrate that VoxHakka significantly outperforms existing publicly available Hakka TTS systems in terms of pronunciation accuracy, tone correctness, and overall naturalness. This work represents a significant advancement in Hakka language technology and provides a valuable resource for language preservation and revitalization efforts.

VoxHakka: A Dialectally Diverse Multi-speaker Text-to-Speech System for Taiwanese Hakka

TL;DR

VoxHakka addresses the data scarcity and pronunciation complexity of Taiwanese Hakka by delivering a zero-shot, dialect-aware TTS system built on YourTTS with dialect and speaker embeddings across six dialects. It collects and cleans a large public-data-derived corpus via ASR-based alignment, forced timing, and silence trimming, then trains a VAE-based end-to-end TTS pipeline with a WaveNet-augmented decoder and a HiFi-GAN vocoder. Objective and CMOS assessments show VoxHakka outperforms existing public Hakka TTS systems in naturalness and tone, with dialect-specific speaker embeddings providing the best naturalness. By releasing the model under CC-BY 4.0, the work offers a valuable resource for language preservation and future research in low-resource, multi-dialect speech technologies.

Abstract

This paper introduces VoxHakka, a text-to-speech (TTS) system designed for Taiwanese Hakka, a critically under-resourced language spoken in Taiwan. Leveraging the YourTTS framework, VoxHakka achieves high naturalness and accuracy and low real-time factor in speech synthesis while supporting six distinct Hakka dialects. This is achieved by training the model with dialect-specific data, allowing for the generation of speaker-aware Hakka speech. To address the scarcity of publicly available Hakka speech corpora, we employed a cost-effective approach utilizing a web scraping pipeline coupled with automatic speech recognition (ASR)-based data cleaning techniques. This process ensured the acquisition of a high-quality, multi-speaker, multi-dialect dataset suitable for TTS training. Subjective listening tests conducted using comparative mean opinion scores (CMOS) demonstrate that VoxHakka significantly outperforms existing publicly available Hakka TTS systems in terms of pronunciation accuracy, tone correctness, and overall naturalness. This work represents a significant advancement in Hakka language technology and provides a valuable resource for language preservation and revitalization efforts.
Paper Structure (13 sections, 3 figures, 4 tables)

This paper contains 13 sections, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Acquisition of Speech for TTS
  3. TTS Model Training
  4. Experiments
  5. Scraped and Cleaned Data
  6. TTS Model Structure of VoxHakka
  7. Evaluation
  8. Systems to be Compared
  9. Our Synthesized Utterances
  10. Comparative Mean Opinion Scores
  11. Results
  12. Conclusion
  13. Acknowledgments

Figures (3)

  • Figure 1: Our procedures of data acquisition and generation. The final data are used for TTS training.
  • Figure 2: Our procedures of TTS model training. The speaker model is pre-trained.
  • Figure 3: Diagram of speech concatenation, where waveforms 1-3 are successive segments that belong to the same utterance, and texts 1-3 are their corresponding texts. Waveform 1+2 denotes the new segment that concatenates waveform 1 and 2, while the text part becomes "text 1,text 2" in series. Note that the comma "," represents a fixed-length pause.