Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SpeechCraft: A Fine-grained Expressive Speech Dataset with Natural Language Description

Zeyu Jin, Jia Jia, Qixin Wang, Kehan Li, Shuoyi Zhou, Songtao Zhou, Xiaoyu Qin, Zhiyong Wu

TL;DR

The paper tackles the need for fine-grained expressive speech data to advance multimodal speech-language models. It introduces an automatic annotation pipeline that fuses acoustic attribute extraction with fine-tuned LLM rewriting to produce highly descriptive, per-audio natural-language captions. The resulting SpeechCraft dataset is bilingual, large-scale (≈2,000 hours, >2 million clips), and designed to support expressive synthesis, style control, and captioning tasks. Empirical results show SpeechCraft improves expressive TTS, enables precise emphasis control, and enhances automated speech-style captioning, demonstrating practical impact for training robust, controllable speech models. This work provides a scalable framework and a high-quality resource that can accelerate development of nuanced, language-grounded speech systems.

Abstract

Speech-language multi-modal learning presents a significant challenge due to the fine nuanced information inherent in speech styles. Therefore, a large-scale dataset providing elaborate comprehension of speech style is urgently needed to facilitate insightful interplay between speech audio and natural language. However, constructing such datasets presents a major trade-off between large-scale data collection and high-quality annotation. To tackle this challenge, we propose an automatic speech annotation system for expressiveness interpretation that annotates in-the-wild speech clips with expressive and vivid human language descriptions. Initially, speech audios are processed by a series of expert classifiers and captioning models to capture diverse speech characteristics, followed by a fine-tuned LLaMA for customized annotation generation. Unlike previous tag/templet-based annotation frameworks with limited information and diversity, our system provides in-depth understandings of speech style through tailored natural language descriptions, thereby enabling accurate and voluminous data generation for large model training. With this system, we create SpeechCraft, a fine-grained bilingual expressive speech dataset. It is distinguished by highly descriptive natural language style prompts, containing approximately 2,000 hours of audio data and encompassing over two million speech clips. Extensive experiments demonstrate that the proposed dataset significantly boosts speech-language task performance in stylist speech synthesis and speech style understanding.

SpeechCraft: A Fine-grained Expressive Speech Dataset with Natural Language Description

TL;DR

The paper tackles the need for fine-grained expressive speech data to advance multimodal speech-language models. It introduces an automatic annotation pipeline that fuses acoustic attribute extraction with fine-tuned LLM rewriting to produce highly descriptive, per-audio natural-language captions. The resulting SpeechCraft dataset is bilingual, large-scale (≈2,000 hours, >2 million clips), and designed to support expressive synthesis, style control, and captioning tasks. Empirical results show SpeechCraft improves expressive TTS, enables precise emphasis control, and enhances automated speech-style captioning, demonstrating practical impact for training robust, controllable speech models. This work provides a scalable framework and a high-quality resource that can accelerate development of nuanced, language-grounded speech systems.

Abstract

Speech-language multi-modal learning presents a significant challenge due to the fine nuanced information inherent in speech styles. Therefore, a large-scale dataset providing elaborate comprehension of speech style is urgently needed to facilitate insightful interplay between speech audio and natural language. However, constructing such datasets presents a major trade-off between large-scale data collection and high-quality annotation. To tackle this challenge, we propose an automatic speech annotation system for expressiveness interpretation that annotates in-the-wild speech clips with expressive and vivid human language descriptions. Initially, speech audios are processed by a series of expert classifiers and captioning models to capture diverse speech characteristics, followed by a fine-tuned LLaMA for customized annotation generation. Unlike previous tag/templet-based annotation frameworks with limited information and diversity, our system provides in-depth understandings of speech style through tailored natural language descriptions, thereby enabling accurate and voluminous data generation for large model training. With this system, we create SpeechCraft, a fine-grained bilingual expressive speech dataset. It is distinguished by highly descriptive natural language style prompts, containing approximately 2,000 hours of audio data and encompassing over two million speech clips. Extensive experiments demonstrate that the proposed dataset significantly boosts speech-language task performance in stylist speech synthesis and speech style understanding.
Paper Structure (32 sections, 5 figures, 7 tables)

This paper contains 32 sections, 5 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Tag-based Speech Datasets
  4. Natural Language Stylistic Datasets
  5. Auto-captioning from Audio to Speech
  6. Descriptive Speech Interpretation
  7. Overview of the Annotation System
  8. Data Preparation
  9. Speech Style Recognition
  10. Rewriting via LLMs
  11. SpeechCraft Dataset
  12. Data Sources
  13. Fine-grained Emphasis Speech Dataset
  14. Validation of the Annotation System
  15. Data Analysis
  16. ...and 17 more sections

Figures (5)

  • Figure 1: System framework of the automatic speech annotation system.
  • Figure 2: Word clouds of top 300 words in English and Chinese parts of SpeechCraft.
  • Figure 3: Sentence length distributions of the speech descriptions generated by different models.
  • Figure 4: Distributions of age, gender and emotion.
  • Figure 5: Mel-spectrogram examples of speech emphasis control. Different stressed words on 'Waitress', 'Wealthy' and 'Woodsman' respectively with the same text instructions.