Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Time-Layer Adaptive Alignment for Speaker Similarity in Flow-Matching Based Zero-Shot TTS

Haoyu Li, Mingyang Han, Yu Xi, Dongxiao Wang, Hankun Wang, Haoxiang Shi, Boyu Li, Jun Song, Bo Zheng, Shuai Wang, Kai Yu

Abstract

Flow-Matching (FM)-based zero-shot text-to-speech (TTS) systems exhibit high-quality speech synthesis and robust generalization capabilities. However, the speaker representation ability of such systems remains underexplored, primarily due to the lack of explicit speaker-specific supervision in the FM framework. To this end, we conduct an empirical analysis of speaker information distribution and reveal its non-uniform allocation across time steps and network layers, underscoring the need for adaptive speaker alignment. Accordingly, we propose Time-Layer Adaptive Speaker Alignment (TLA-SA), a strategy that enhances speaker consistency by jointly leveraging temporal and hierarchical variations. Experimental results show that TLA-SA substantially improves speaker similarity over baseline systems on both research- and industrial-scale datasets and generalizes well across diverse model architectures, including decoder-only language model (LM)-based and free TTS systems. A demo is provided.

Time-Layer Adaptive Alignment for Speaker Similarity in Flow-Matching Based Zero-Shot TTS

Abstract

Flow-Matching (FM)-based zero-shot text-to-speech (TTS) systems exhibit high-quality speech synthesis and robust generalization capabilities. However, the speaker representation ability of such systems remains underexplored, primarily due to the lack of explicit speaker-specific supervision in the FM framework. To this end, we conduct an empirical analysis of speaker information distribution and reveal its non-uniform allocation across time steps and network layers, underscoring the need for adaptive speaker alignment. Accordingly, we propose Time-Layer Adaptive Speaker Alignment (TLA-SA), a strategy that enhances speaker consistency by jointly leveraging temporal and hierarchical variations. Experimental results show that TLA-SA substantially improves speaker similarity over baseline systems on both research- and industrial-scale datasets and generalizes well across diverse model architectures, including decoder-only language model (LM)-based and free TTS systems. A demo is provided.

Paper Structure

This paper contains 16 sections, 7 equations, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Flow-Matching based zero-shot TTS
  4. Speaker distribution in the FM-based TTS
  5. Time and layer adaptive speaker alignment
  6. Experimental Setups
  7. Datasets
  8. Model structures and training configurations
  9. Evaluation Metrics
  10. Results and Analysis
  11. TLA-SA on large-scale dataset
  12. TLA-SA on high-quality LibriTTS
  13. Performance on different model structures
  14. Subjective evaluation on speaker similarity
  15. Conclusion
  16. ...and 1 more sections

Figures (5)

  • Figure 1: Overview of TLA-SA. A U-Net-based FM model is illustrated, and similar integration patterns apply to other architectures. TLA-SA enforces speaker alignment across intermediate Transformer blocks, incorporating supervision on both timestep and model layer dimensions.
  • Figure 2: Correlation between speaker similarity and CKNNA. Each point denotes a checkpoint at a distinct training step.
  • Figure 3: CKNNA varies across time and layers on the converged checkpoint.
  • Figure 4: Sim-WavLM across steps on Seed-TTS test-zh.
  • Figure 5: Heatmap of adaptive $\mathbf{w}$ at different training stage.