Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CAST-TTS: A Simple Cross-Attention Framework for Unified Timbre Control in TTS

Zihao Zheng, Wen Wu, Chao Zhang, Mengyue Wu, Xuenan Xu

Abstract

Current Text-to-Speech (TTS) systems typically use separate models for speech-prompted and text-prompted timbre control. While unifying both control signals into a single model is desirable, the challenge of cross-modal alignment often results in overly complex architectures and training objective. To address this challenge, we propose CAST-TTS, a simple yet effective framework for unified timbre control. Features are extracted from speech prompts and text prompts using pre-trained encoders. The multi-stage training strategy efficiently aligns the speech and projected text representations within a shared embedding space. A single cross-attention mechanism then allows the model to use either of these representations to control the timbre. Extensive experiments validate that the unified cross-attention mechanism is critical for achieving high-quality synthesis. CAST-TTS achieves performance comparable to specialized single-input models while operating within a unified architecture. The demo page can be accessed at https://HiRookie9.github.io/CAST-TTS-Page.

CAST-TTS: A Simple Cross-Attention Framework for Unified Timbre Control in TTS

Abstract

Current Text-to-Speech (TTS) systems typically use separate models for speech-prompted and text-prompted timbre control. While unifying both control signals into a single model is desirable, the challenge of cross-modal alignment often results in overly complex architectures and training objective. To address this challenge, we propose CAST-TTS, a simple yet effective framework for unified timbre control. Features are extracted from speech prompts and text prompts using pre-trained encoders. The multi-stage training strategy efficiently aligns the speech and projected text representations within a shared embedding space. A single cross-attention mechanism then allows the model to use either of these representations to control the timbre. Extensive experiments validate that the unified cross-attention mechanism is critical for achieving high-quality synthesis. CAST-TTS achieves performance comparable to specialized single-input models while operating within a unified architecture. The demo page can be accessed at https://HiRookie9.github.io/CAST-TTS-Page.
Paper Structure (17 sections, 2 equations, 1 figure, 3 tables)

This paper contains 17 sections, 2 equations, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. CAST-TTS
  3. Architecture
  4. Training
  5. Inference
  6. Experiments
  7. Datasets
  8. Experimental Setup
  9. Evaluation
  10. Results
  11. Generation Performance
  12. Ablation Study
  13. Speaker Features
  14. Fusion Mechanism
  15. Training Strategy
  16. ...and 2 more sections

Figures (1)

  • Figure 1: An overview of CAST-TTS. The timbre encoder converts speech or text prompts into timbre embeddings, which condition the synthesis model through a cross-attention mechanism.