Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mismatch Aware Guidance for Robust Emotion Control in Auto-Regressive TTS Models

Yizhou Peng, Yukun Ma, Chong Zhang, Yi-Wen Chao, Chongjia Ni, Bin Ma

TL;DR

This work addresses mismatches between emotional style prompts and semantic content in auto-regressive TTS. It analyzes classifier-free guidance (CFG) in AR TTS and introduces an adaptive, mismatch-aware scheme that leverages signals from large language models and natural language inference to adjust CFG strength. The key contribution is the Semantic Mismatch Guided CFG (SMG-CFG), along with a random-condition variant, CFG-Filter, and a few-shot fine-tuning protocol on CosyVoice2, evaluated on the TextrolSpeech dataset. Results show improved emotional expressiveness without sacrificing intelligibility or naturalness, enabling robust, fine-grained control of expressive speech in AR TTS for real-world applications.

Abstract

While Text-to-Speech (TTS) systems can achieve fine-grained control over emotional expression via natural language prompts, a significant challenge emerges when the desired emotion (style prompt) conflicts with the semantic content of the text. This mismatch often results in unnatural-sounding speech, undermining the goal of achieving fine-grained emotional control. Classifier-Free Guidance (CFG) is a key technique for enhancing prompt alignment; however, its application to auto-regressive (AR) TTS models remains underexplored, which can lead to degraded audio quality. This paper directly addresses the challenge of style-content mismatch in AR TTS models by proposing an adaptive CFG scheme that adjusts to different levels of the detected mismatch, as measured using large language models or natural language inference models. This solution is based on a comprehensive analysis of CFG's impact on emotional expressiveness in state-of-the-art AR TTS models. Our results demonstrate that the proposed adaptive CFG scheme improves the emotional expressiveness of the AR TTS model while maintaining audio quality and intelligibility.

Mismatch Aware Guidance for Robust Emotion Control in Auto-Regressive TTS Models

TL;DR

This work addresses mismatches between emotional style prompts and semantic content in auto-regressive TTS. It analyzes classifier-free guidance (CFG) in AR TTS and introduces an adaptive, mismatch-aware scheme that leverages signals from large language models and natural language inference to adjust CFG strength. The key contribution is the Semantic Mismatch Guided CFG (SMG-CFG), along with a random-condition variant, CFG-Filter, and a few-shot fine-tuning protocol on CosyVoice2, evaluated on the TextrolSpeech dataset. Results show improved emotional expressiveness without sacrificing intelligibility or naturalness, enabling robust, fine-grained control of expressive speech in AR TTS for real-world applications.

Abstract

While Text-to-Speech (TTS) systems can achieve fine-grained control over emotional expression via natural language prompts, a significant challenge emerges when the desired emotion (style prompt) conflicts with the semantic content of the text. This mismatch often results in unnatural-sounding speech, undermining the goal of achieving fine-grained emotional control. Classifier-Free Guidance (CFG) is a key technique for enhancing prompt alignment; however, its application to auto-regressive (AR) TTS models remains underexplored, which can lead to degraded audio quality. This paper directly addresses the challenge of style-content mismatch in AR TTS models by proposing an adaptive CFG scheme that adjusts to different levels of the detected mismatch, as measured using large language models or natural language inference models. This solution is based on a comprehensive analysis of CFG's impact on emotional expressiveness in state-of-the-art AR TTS models. Our results demonstrate that the proposed adaptive CFG scheme improves the emotional expressiveness of the AR TTS model while maintaining audio quality and intelligibility.
Paper Structure (21 sections, 4 equations, 2 figures, 4 tables)

This paper contains 21 sections, 4 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Classifier-Free Guidance (CFG)
  4. Standard CFG
  5. CFG with Top-k Filtering
  6. Guidance with a Random Condition
  7. Semantic Mismatch Guided CFG (SMG-CFG)
  8. Experimental Setup
  9. Dataset
  10. TTS model
  11. CFG implementation on CosyVoice2
  12. Few-shot fine-tuning on CosyVoice2
  13. Evaluation Methods
  14. Emotional Expressiveness
  15. Overall Speech Quality
  16. ...and 6 more sections

Figures (2)

  • Figure 1: Proposed semantic mismatch guided CFG strategy for an AR TTS model. The CFG requires two types of inputs: conditional and unconditional. The conditional input consists of a style prompt and the target text. The unconditional input only includes the target text. For the guidance scale of CFG, we implement two models to discriminate the semantic mismatch between the Style Prompt and the Target Text, and we map the mismatch level into a set of specific scale values to guide the CFG inference.
  • Figure 2: The ER ACC. and ASR WER evaluation on the synthesized speech vs. increasing CFG Scales. The full lines represent the performance of our three systems that apply CFG. Drop_Prompt means that the unconditioned drop the style, while Drop_Target represents the unconditioned drop the target text. The black dashed line represents the baseline system without CFG, while the blue and green dashed lines correspond to our proposed SMG-CFG based on Drop_Prompt and Drop_Prompt_Filter settings, respectively.