Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Do You Hear What I Mean? Quantifying the Instruction-Perception Gap in Instruction-Guided Expressive Text-To-Speech Systems

Yi-Cheng Lin, Huang-Cheng Chou, Tzu-Chieh Wei, Kuan-Yu Chen, Hung-yi Lee

Abstract

Instruction-guided text-to-speech (ITTS) enables users to control speech generation through natural language prompts, offering a more intuitive interface than traditional TTS. However, the alignment between user style instructions and listener perception remains largely unexplored. This work first presents a perceptual analysis of ITTS controllability across two expressive dimensions (adverbs of degree and graded emotion intensity) and collects human ratings on speaker age and word-level emphasis attributes. To comprehensively reveal the instruction-perception gap, we provide a data collection with large-scale human evaluations, named Expressive VOice Control (E-VOC) corpus. Furthermore, we reveal that (1) gpt-4o-mini-tts is the most reliable ITTS model with great alignment between instruction and generated utterances across acoustic dimensions. (2) The 5 analyzed ITTS systems tend to generate Adult voices even when the instructions ask to use child or Elderly voices. (3) Fine-grained control remains a major challenge, indicating that most ITTS systems have substantial room for improvement in interpreting slightly different attribute instructions.

Do You Hear What I Mean? Quantifying the Instruction-Perception Gap in Instruction-Guided Expressive Text-To-Speech Systems

Abstract

Instruction-guided text-to-speech (ITTS) enables users to control speech generation through natural language prompts, offering a more intuitive interface than traditional TTS. However, the alignment between user style instructions and listener perception remains largely unexplored. This work first presents a perceptual analysis of ITTS controllability across two expressive dimensions (adverbs of degree and graded emotion intensity) and collects human ratings on speaker age and word-level emphasis attributes. To comprehensively reveal the instruction-perception gap, we provide a data collection with large-scale human evaluations, named Expressive VOice Control (E-VOC) corpus. Furthermore, we reveal that (1) gpt-4o-mini-tts is the most reliable ITTS model with great alignment between instruction and generated utterances across acoustic dimensions. (2) The 5 analyzed ITTS systems tend to generate Adult voices even when the instructions ask to use child or Elderly voices. (3) Fine-grained control remains a major challenge, indicating that most ITTS systems have substantial room for improvement in interpreting slightly different attribute instructions.

Paper Structure

This paper contains 21 sections, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related works and Background
  3. ITTS Systems and Selection
  4. ITTS Evaluation Methods
  5. Evaluation Framework
  6. Control Dimension
  7. Proposed Control Dimensions
  8. Other Control Dimensions
  9. Evaluation Metrics
  10. Objective Measures
  11. Subjective Measures
  12. Human Annotation Collection & the E-VOC dataset
  13. Transcripts Generation
  14. Annotation and Quality Control
  15. Experimental Results and Analyses
  16. ...and 6 more sections

Figures (2)

  • Figure 1: Loudness (LUFS), pitch (Hz), and speaking rate (words/s) across ITTS models for Task I. Adverbs of Degree.
  • Figure 2: Averaged perceptual emotion intensity of ITTS models across 4 emotions (e.g., Angry, Happy, Surprised; Sad), analyzed by Task I. Adverbs of Degree (top row) and Task II. Emotion–Intensity Adjective (bottom row). The figure shared the same legend with Figure \ref{['fig:tts_three_in_one']}.