Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring speech style spaces with language models: Emotional TTS without emotion labels

Shreeram Suresh Chandra, Zongyang Du, Berrak Sisman

TL;DR

This paper tackles the challenge of emotional text-to-speech (E-TTS) without relying on scarce emotion labels. It introduces TEMOTTS, a two-stage framework that constructs an unsupervised emotional style space via global style tokens (GST) and aligns emotional text embeddings to this space using a language-model-driven adaptation, enabling inference from text alone. Stage I learns a GST-based style space within a FastSpeech2 backbone, while Stage II prunes emotion-rich text with DistilRoBERTa, extracts emotion embeddings, and trains an adaptation to map them to GST weights. Evaluations show TEMOTTS improves intelligibility, emotional accuracy, and naturalness compared to label-dependent baselines, demonstrating a robust link between linguistic content and expressive prosody and enabling scalable, text-only emotional synthesis.

Abstract

Many frameworks for emotional text-to-speech (E-TTS) rely on human-annotated emotion labels that are often inaccurate and difficult to obtain. Learning emotional prosody implicitly presents a tough challenge due to the subjective nature of emotions. In this study, we propose a novel approach that leverages text awareness to acquire emotional styles without the need for explicit emotion labels or text prompts. We present TEMOTTS, a two-stage framework for E-TTS that is trained without emotion labels and is capable of inference without auxiliary inputs. Our proposed method performs knowledge transfer between the linguistic space learned by BERT and the emotional style space constructed by global style tokens. Our experimental results demonstrate the effectiveness of our proposed framework, showcasing improvements in emotional accuracy and naturalness. This is one of the first studies to leverage the emotional correlation between spoken content and expressive delivery for emotional TTS.

Exploring speech style spaces with language models: Emotional TTS without emotion labels

TL;DR

This paper tackles the challenge of emotional text-to-speech (E-TTS) without relying on scarce emotion labels. It introduces TEMOTTS, a two-stage framework that constructs an unsupervised emotional style space via global style tokens (GST) and aligns emotional text embeddings to this space using a language-model-driven adaptation, enabling inference from text alone. Stage I learns a GST-based style space within a FastSpeech2 backbone, while Stage II prunes emotion-rich text with DistilRoBERTa, extracts emotion embeddings, and trains an adaptation to map them to GST weights. Evaluations show TEMOTTS improves intelligibility, emotional accuracy, and naturalness compared to label-dependent baselines, demonstrating a robust link between linguistic content and expressive prosody and enabling scalable, text-only emotional synthesis.

Abstract

Many frameworks for emotional text-to-speech (E-TTS) rely on human-annotated emotion labels that are often inaccurate and difficult to obtain. Learning emotional prosody implicitly presents a tough challenge due to the subjective nature of emotions. In this study, we propose a novel approach that leverages text awareness to acquire emotional styles without the need for explicit emotion labels or text prompts. We present TEMOTTS, a two-stage framework for E-TTS that is trained without emotion labels and is capable of inference without auxiliary inputs. Our proposed method performs knowledge transfer between the linguistic space learned by BERT and the emotional style space constructed by global style tokens. Our experimental results demonstrate the effectiveness of our proposed framework, showcasing improvements in emotional accuracy and naturalness. This is one of the first studies to leverage the emotional correlation between spoken content and expressive delivery for emotional TTS.
Paper Structure (16 sections, 5 figures, 1 table)

This paper contains 16 sections, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related work
  3. Towards using fewer emotion labels for E-TTS
  4. Text-aware TTS
  5. TEMOTTS
  6. Stage I: Constructing an emotional style space
  7. Stage II: Exploration of style space with language model
  8. Inference
  9. Experiments
  10. Experimental setup
  11. Choice of baselines:
  12. Choice of number of style tokens:
  13. Objective evaluation
  14. Comparing pitch contours:
  15. Subjective evaluation
  16. ...and 1 more sections

Figures (5)

  • Figure 1: Proposed TEMOTTS framework with stage I and stage II. Red modules contain trainable weights and purple modules contain fixed weights; CE loss represents Categorical Cross Entropy loss.
  • Figure 2: Visualization of emotional style space and emotional text embeddings space with a conceptual link.
  • Figure 3: Inference strategy of TEMOTTS.
  • Figure 4: SER confusion matrices of 4 emotion categories.
  • Figure 5: Pitch contours of speech synthesized by TEMOTTS and Fastspeech2 along with ground truth.