Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VoiceSculptor: Your Voice, Designed By You

Jingbin Hu, Huakang Chen, Linhan Ma, Dake Guo, Qirui Zhan, Wenhao Li, Haoyu Zhang, Kangxiang Xia, Ziyu Zhang, Wenjie Tian, Chengyou Wang, Jinrui Liang, Shuhan Guo, Zihang Yang, Bengu Wu, Binbin Zhang, Pengcheng Zhu, Pengyuan Xie, Chuan Xie, Qiang Zhang, Jie Liu, Lei Xie

TL;DR

VoiceSculptor tackles the bottleneck of fine-grained, instruction-driven voice control in open TTS by integrating a CoT-based attribute reasoning framework with retrieval-grounded instruction generalization. The system pairs a LLaSA-3B–based voice design module with CosyVoice2 for high-fidelity cloning, using XCodec2 tokenization to operate in an autoregressive, text-and-audio token space. Key contributions include a large-scale, multi-level annotated data pipeline, CoT-based attribute tokens with token dropout, and a retrieval-augmented grounding mechanism that enhances robustness to out-of-domain instructions. Empirical results on InstructTTSEval-Zh demonstrate state-of-the-art open-source performance, with scaling and ablation studies validating the effectiveness of CoT reasoning, text supervision, and RAG in achieving precise, controllable voice synthesis suitable for practical downstream deployment.

Abstract

Despite rapid progress in text-to-speech (TTS), open-source systems still lack truly instruction-following, fine-grained control over core speech attributes (e.g., pitch, speaking rate, age, emotion, and style). We present VoiceSculptor, an open-source unified system that bridges this gap by integrating instruction-based voice design and high-fidelity voice cloning in a single framework. It generates controllable speaker timbre directly from natural-language descriptions, supports iterative refinement via Retrieval-Augmented Generation (RAG), and provides attribute-level edits across multiple dimensions. The designed voice is then rendered into a prompt waveform and fed into a cloning model to enable high-fidelity timbre transfer for downstream speech synthesis. VoiceSculptor achieves open-source state-of-the-art (SOTA) on InstructTTSEval-Zh, and is fully open-sourced, including code and pretrained models, to advance reproducible instruction-controlled TTS research.

VoiceSculptor: Your Voice, Designed By You

TL;DR

VoiceSculptor tackles the bottleneck of fine-grained, instruction-driven voice control in open TTS by integrating a CoT-based attribute reasoning framework with retrieval-grounded instruction generalization. The system pairs a LLaSA-3B–based voice design module with CosyVoice2 for high-fidelity cloning, using XCodec2 tokenization to operate in an autoregressive, text-and-audio token space. Key contributions include a large-scale, multi-level annotated data pipeline, CoT-based attribute tokens with token dropout, and a retrieval-augmented grounding mechanism that enhances robustness to out-of-domain instructions. Empirical results on InstructTTSEval-Zh demonstrate state-of-the-art open-source performance, with scaling and ablation studies validating the effectiveness of CoT reasoning, text supervision, and RAG in achieving precise, controllable voice synthesis suitable for practical downstream deployment.

Abstract

Despite rapid progress in text-to-speech (TTS), open-source systems still lack truly instruction-following, fine-grained control over core speech attributes (e.g., pitch, speaking rate, age, emotion, and style). We present VoiceSculptor, an open-source unified system that bridges this gap by integrating instruction-based voice design and high-fidelity voice cloning in a single framework. It generates controllable speaker timbre directly from natural-language descriptions, supports iterative refinement via Retrieval-Augmented Generation (RAG), and provides attribute-level edits across multiple dimensions. The designed voice is then rendered into a prompt waveform and fed into a cloning model to enable high-fidelity timbre transfer for downstream speech synthesis. VoiceSculptor achieves open-source state-of-the-art (SOTA) on InstructTTSEval-Zh, and is fully open-sourced, including code and pretrained models, to advance reproducible instruction-controlled TTS research.
Paper Structure (15 sections, 5 figures, 6 tables)

This paper contains 15 sections, 5 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Architecture
  3. Overview
  4. Data Processing Pipeline
  5. CoT–based Fine-grained Attribute Modeling
  6. Retrieval-augmented Instruction Generalization
  7. Experiments
  8. Evaluation On InstructTTSEval-Zh Benchmark
  9. Scaling Study On Model Size And Data Size
  10. Ablation Study On CoT-based Fine-grained Attribute Tokens
  11. Ablation Study Of Text Cross-Entropy Loss
  12. Ablation Study Of RAG
  13. Conclusion
  14. Ethics Statement
  15. Appendix

Figures (5)

  • Figure 1: The overview of VoiceSculptor, which is composed of two core components: voice design and voice clone.
  • Figure 2: The data pipeline of building VoiceSculptor.
  • Figure 3: Validation loss curves for the ablation study on CoT and model scale.
  • Figure 4: Validation loss curves of the SFT stage under different data configurations.
  • Figure :