Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ManchuTTS: Towards High-Quality Manchu Speech Synthesis via Flow Matching and Hierarchical Text Representation

Suhua Wang, Zifan Wang, Xiaoxin Sun, D. J. Wang, Zhanbo Liu, Xin Li

TL;DR

This work tackles the data scarcity and phonological complexity of Manchu by introducing ManchuTTS, a data-efficient TTS framework built on a three-layer hierarchical text representation and a conditional flow-matching generator. It integrates a three-layer cross-modal attention mechanism and a hierarchical contrastive loss to achieve implicit, multi-granular alignment between text and speech, enabling non-autoregressive, diffusion-transformer-based synthesis. Empirical results on the first public Manchu TTS dataset show a MOS of 4.52 with 5.2 hours of training data, with ablations confirming substantial gains from phoneme, syllable, and prosody guidance (AWPA up to 31%, prosodic naturalness up to 27%). The approach also demonstrates data-efficiency, competitive zero-shot cross-language transfer to Ewenki, and deployment-readiness on consumer hardware, offering a practical pathway for preserving and revitalizing endangered languages.

Abstract

As an endangered language, Manchu presents unique challenges for speech synthesis, including severe data scarcity and strong phonological agglutination. This paper proposes ManchuTTS(Manchu Text to Speech), a novel approach tailored to Manchu's linguistic characteristics. To handle agglutination, this method designs a three-tier text representation (phoneme, syllable, prosodic) and a cross-modal hierarchical attention mechanism for multi-granular alignment. The synthesis model integrates deep convolutional networks with a flow-matching Transformer, enabling efficient, non-autoregressive generation. This method further introduce a hierarchical contrastive loss to guide structured acoustic-linguistic correspondence. To address low-resource constraints, This method construct the first Manchu TTS dataset and employ a data augmentation strategy. Experiments demonstrate that ManchuTTS attains a MOS of 4.52 using a 5.2-hour training subset derived from our full 6.24-hour annotated corpus, outperforming all baseline models by a notable margin. Ablations confirm hierarchical guidance improves agglutinative word pronunciation accuracy (AWPA) by 31% and prosodic naturalness by 27%.

ManchuTTS: Towards High-Quality Manchu Speech Synthesis via Flow Matching and Hierarchical Text Representation

TL;DR

This work tackles the data scarcity and phonological complexity of Manchu by introducing ManchuTTS, a data-efficient TTS framework built on a three-layer hierarchical text representation and a conditional flow-matching generator. It integrates a three-layer cross-modal attention mechanism and a hierarchical contrastive loss to achieve implicit, multi-granular alignment between text and speech, enabling non-autoregressive, diffusion-transformer-based synthesis. Empirical results on the first public Manchu TTS dataset show a MOS of 4.52 with 5.2 hours of training data, with ablations confirming substantial gains from phoneme, syllable, and prosody guidance (AWPA up to 31%, prosodic naturalness up to 27%). The approach also demonstrates data-efficiency, competitive zero-shot cross-language transfer to Ewenki, and deployment-readiness on consumer hardware, offering a practical pathway for preserving and revitalizing endangered languages.

Abstract

As an endangered language, Manchu presents unique challenges for speech synthesis, including severe data scarcity and strong phonological agglutination. This paper proposes ManchuTTS(Manchu Text to Speech), a novel approach tailored to Manchu's linguistic characteristics. To handle agglutination, this method designs a three-tier text representation (phoneme, syllable, prosodic) and a cross-modal hierarchical attention mechanism for multi-granular alignment. The synthesis model integrates deep convolutional networks with a flow-matching Transformer, enabling efficient, non-autoregressive generation. This method further introduce a hierarchical contrastive loss to guide structured acoustic-linguistic correspondence. To address low-resource constraints, This method construct the first Manchu TTS dataset and employ a data augmentation strategy. Experiments demonstrate that ManchuTTS attains a MOS of 4.52 using a 5.2-hour training subset derived from our full 6.24-hour annotated corpus, outperforming all baseline models by a notable margin. Ablations confirm hierarchical guidance improves agglutinative word pronunciation accuracy (AWPA) by 31% and prosodic naturalness by 27%.
Paper Structure (17 sections, 10 equations, 8 figures, 7 tables)

This paper contains 17 sections, 10 equations, 8 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Three-Layer Cross-Modal Attention Mechanism
  4. Hierarchical Text Representation
  5. Hierarchical Contrastive Alignment Loss
  6. Model Innovations and Comparison with Existing Methods
  7. Experiments
  8. Experimental Setup
  9. Experimental Results and Analysis
  10. Main Results
  11. Ablation Study of Hierarchical Text Feature Guidance Mechanism
  12. Sensitivity Analysis of Data Scale
  13. Cross-Language Transfer and Zero-Shot Generalization
  14. Linguistic Verification
  15. Computational Efficiency and Deployment Analysis
  16. ...and 2 more sections

Figures (8)

  • Figure 1: Qing Dynasty documents that have not been translated or interpreted to date.
  • Figure 2: Annotation example of Manchu speech using the Praat tool.
  • Figure 3: Overall Pipeline of ManchuTTS Framework. The system encodes target text into three-level linguistic features, processes them through an 8-layer Diffusion Transformer (DiT) backbone, and generates speech via conditional flow matching.
  • Figure 4: Radar chart of multi-dimensional performance for main Manchu TTS experiments (ManchuTTS vs. baseline models).
  • Figure 5: Impact of three-level text feature guidance on MOS, AWPA, and prosodic naturalness. X-axis: Configuration levels ("Phoneme", "+Syllable", "+Prosody"), representing the incremental guidance conditions added step by step. Y-axis: 1. MOS (Mean Opinion Score, 1–5, the higher the better); 2. AWPA (%, the higher the better); 3. Prosodic naturalness (%, the higher the better).
  • ...and 3 more figures