Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MamTra: A Hybrid Mamba-Transformer Backbone for Speech Synthesis

Tan Dat Nguyen, Sangmin Bae, Joon Son Chung, Ji-Hoon Kim

Abstract

Despite the remarkable quality of LLM-based text-to-speech systems, their reliance on autoregressive Transformers leads to quadratic computational complexity, which severely limits practical applications. Linear-time alternatives, notably Mamba, offer a potential remedy; however, they often sacrifice the global context essential for expressive synthesis. In this paper, we propose MamTra, an interleaved Mamba-Transformer framework designed to leverage the advantages of Mamba's efficiency and Transformers' modeling capability. We also introduce novel knowledge transfer strategies to distill insights from a pretrained Transformer into our hybrid architecture, thereby bypassing the prohibitive costs of training from scratch. Systematic experiments identify the optimal hybrid configuration, and demonstrate that MamTra reduces inference VRAM usage by up to 34% without compromising speech fidelity - even trained on only 2% of the original training dataset. Audio samples are available at https://mamtratts.github.io.

MamTra: A Hybrid Mamba-Transformer Backbone for Speech Synthesis

Abstract

Despite the remarkable quality of LLM-based text-to-speech systems, their reliance on autoregressive Transformers leads to quadratic computational complexity, which severely limits practical applications. Linear-time alternatives, notably Mamba, offer a potential remedy; however, they often sacrifice the global context essential for expressive synthesis. In this paper, we propose MamTra, an interleaved Mamba-Transformer framework designed to leverage the advantages of Mamba's efficiency and Transformers' modeling capability. We also introduce novel knowledge transfer strategies to distill insights from a pretrained Transformer into our hybrid architecture, thereby bypassing the prohibitive costs of training from scratch. Systematic experiments identify the optimal hybrid configuration, and demonstrate that MamTra reduces inference VRAM usage by up to 34% without compromising speech fidelity - even trained on only 2% of the original training dataset. Audio samples are available at https://mamtratts.github.io.
Paper Structure (14 sections, 5 equations, 6 figures, 4 tables)

This paper contains 14 sections, 5 equations, 6 figures, 4 tables.

Table of Contents

  1. Introduction
  2. MamTra Hybrid Architecture
  3. Preliminaries
  4. Hybridization Strategy
  5. Linearization and Structural Mapping
  6. Multi-Level Distillation
  7. Experimental Setup
  8. Analysis
  9. Efficiency and Complexity Analysis
  10. Efficiency-Quality Tradeoff
  11. Sensitivity Under Length-Stress Conditions
  12. Training Dynamics and Loss Contributions
  13. Conclusion
  14. Generative AI Use Disclosure

Figures (6)

  • Figure 1: Overview of the hybrid Mamba-Transformer configurations for speech synthesis. Selective Transformer-to-Mamba layer transfer reduces training cost and accelerates convergence, while performance is recovered via knowledge distillation using less than 2% of the teacher’s English training data.
  • Figure 2: Following the alignment between Eq. \ref{['eq:ssm_discrete']} and Eq. \ref{['eq:closed_form']}, projection weights for $C$, $B$, and $x$ in Mamba are initialized with Transformer's $Q$, $K$, and $V$ projection weights, respectively.
  • Figure 3: Memory usage (GB) on Seed-TTS-eval (using NVIDIA A6000). MamTra 1:1 reduces VRAM by 34% and 17% compared to CosyVoice 2 and Zonos-v0.1, respectively.
  • Figure 4: Cross-entropy (CE) loss and Word Error Rate (WER) on Seed-TTS-eval for hybrid Mamba-Transformer variants after 15 training epochs on LibriTTS (0.5k h). CE loss exhibits a consistent correlation with WER across different ratios and placement strategies.
  • Figure 5: Cache size growth in the hybrid model, where the sequence-length dependency scales with the number of remaining Transformer layers.
  • ...and 1 more figures