Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

StoryTTS: A Highly Expressive Text-to-Speech Dataset with Rich Textual Expressiveness Annotations

Sen Liu, Yiwei Guo, Xie Chen, Kai Yu

TL;DR

StoryTTS introduces a high-quality Mandarin storytelling TTS dataset with rich acoustic expressiveness and a comprehensive textual expressiveness annotation framework across five dimensions. It combines meticulous data curation, manual correction, punctuation refinement, and LLM-driven labeling to produce 60.9 hours of speech and 33,108 aligned speech-text pairs across 160 chapters. An expressiveness encoder augments a VQTTS baseline, and experiments show that integrating all textual expressiveness labels yields notable improvements in objective metrics (MCD, log-F0 RMSE) and subjective MOS, validating the value of text-driven expressiveness for expressive TTS. The work highlights the practical potential of aligning textual annotation with acoustic synthesis and suggests future integration of textual and acoustic expressiveness to further enhance naturalness and expressiveness in TTS systems.

Abstract

While acoustic expressiveness has long been studied in expressive text-to-speech (ETTS), the inherent expressiveness in text lacks sufficient attention, especially for ETTS of artistic works. In this paper, we introduce StoryTTS, a highly ETTS dataset that contains rich expressiveness both in acoustic and textual perspective, from the recording of a Mandarin storytelling show. A systematic and comprehensive labeling framework is proposed for textual expressiveness. We analyze and define speech-related textual expressiveness in StoryTTS to include five distinct dimensions through linguistics, rhetoric, etc. Then we employ large language models and prompt them with a few manual annotation examples for batch annotation. The resulting corpus contains 61 hours of consecutive and highly prosodic speech equipped with accurate text transcriptions and rich textual expressiveness annotations. Therefore, StoryTTS can aid future ETTS research to fully mine the abundant intrinsic textual and acoustic features. Experiments are conducted to validate that TTS models can generate speech with improved expressiveness when integrating with the annotated textual labels in StoryTTS.

StoryTTS: A Highly Expressive Text-to-Speech Dataset with Rich Textual Expressiveness Annotations

TL;DR

StoryTTS introduces a high-quality Mandarin storytelling TTS dataset with rich acoustic expressiveness and a comprehensive textual expressiveness annotation framework across five dimensions. It combines meticulous data curation, manual correction, punctuation refinement, and LLM-driven labeling to produce 60.9 hours of speech and 33,108 aligned speech-text pairs across 160 chapters. An expressiveness encoder augments a VQTTS baseline, and experiments show that integrating all textual expressiveness labels yields notable improvements in objective metrics (MCD, log-F0 RMSE) and subjective MOS, validating the value of text-driven expressiveness for expressive TTS. The work highlights the practical potential of aligning textual annotation with acoustic synthesis and suggests future integration of textual and acoustic expressiveness to further enhance naturalness and expressiveness in TTS systems.

Abstract

While acoustic expressiveness has long been studied in expressive text-to-speech (ETTS), the inherent expressiveness in text lacks sufficient attention, especially for ETTS of artistic works. In this paper, we introduce StoryTTS, a highly ETTS dataset that contains rich expressiveness both in acoustic and textual perspective, from the recording of a Mandarin storytelling show. A systematic and comprehensive labeling framework is proposed for textual expressiveness. We analyze and define speech-related textual expressiveness in StoryTTS to include five distinct dimensions through linguistics, rhetoric, etc. Then we employ large language models and prompt them with a few manual annotation examples for batch annotation. The resulting corpus contains 61 hours of consecutive and highly prosodic speech equipped with accurate text transcriptions and rich textual expressiveness annotations. Therefore, StoryTTS can aid future ETTS research to fully mine the abundant intrinsic textual and acoustic features. Experiments are conducted to validate that TTS models can generate speech with improved expressiveness when integrating with the annotated textual labels in StoryTTS.
Paper Structure (20 sections, 2 figures, 4 tables)

This paper contains 20 sections, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. StoryTTS
  3. Data selecting and retrieval
  4. Audio quality analysis
  5. Speech segmentation and automatic recognition
  6. Manual correction of recognition errors
  7. Punctuation enhancement
  8. LLM-driven expressiveness annotation
  9. Exploring textual expressiveness
  10. Batch annotation via LLMs
  11. Experiments and results
  12. Model Architecture
  13. Baseline Model
  14. Expressiveness Encoder
  15. Experimental setup
  16. ...and 5 more sections

Figures (2)

  • Figure 1: An Illustration of Expressiveness Annotation Using LLMs.
  • Figure 2: Top 6 role types in StoryTTS.