VowelPrompt: Hearing Speech Emotions from Text via Vowel-level Prosodic Augmentation

TL;DR

VowelPrompt tackles the challenge of emotion recognition from speech by injecting fine-grained vowel-level prosodic cues into large language model reasoning. It extracts $F_0$ level and slope, $F_0$ variation, RMS energy level and variation, and vowel duration from time-aligned vowel segments, normalizes and discretizes them into textual descriptors, and appends them to transcripts to enable joint lexical-prosodic reasoning. A two-stage training pipeline—supervised fine-tuning (SFT) followed by Reinforcement Learning with Verifiable Reward (RLVR) via Group Relative Policy Optimization (GRPO)—improves reasoning quality and output structure while maintaining generalization. Across five SER benchmarks and multilingual settings, VowelPrompt consistently outperforms transcript-only and sentence-level baselines in zero-shot, fine-tuned, cross-domain, and multilingual scenarios, while offering interpretable, vowel-grounded explanations of its decisions.

Abstract

Emotion recognition in speech presents a complex multimodal challenge, requiring comprehension of both linguistic content and vocal expressivity, particularly prosodic features such as fundamental frequency, intensity, and temporal dynamics. Although large language models (LLMs) have shown promise in reasoning over textual transcriptions for emotion recognition, they typically neglect fine-grained prosodic information, limiting their effectiveness and interpretability. In this work, we propose VowelPrompt, a linguistically grounded framework that augments LLM-based emotion recognition with interpretable, fine-grained vowel-level prosodic cues. Drawing on phonetic evidence that vowels serve as primary carriers of affective prosody, VowelPrompt extracts pitch-, energy-, and duration-based descriptors from time-aligned vowel segments, and converts these features into natural language descriptions for better interpretability. Such a design enables LLMs to jointly reason over lexical semantics and fine-grained prosodic variation. Moreover, we adopt a two-stage adaptation procedure comprising supervised fine-tuning (SFT) followed by Reinforcement Learning with Verifiable Reward (RLVR), implemented via Group Relative Policy Optimization (GRPO), to enhance reasoning capability, enforce structured output adherence, and improve generalization across domains and speaker variations. Extensive evaluations across diverse benchmark datasets demonstrate that VowelPrompt consistently outperforms state-of-the-art emotion recognition methods under zero-shot, fine-tuned, cross-domain, and cross-linguistic conditions, while enabling the generation of interpretable explanations that are jointly grounded in contextual semantics and fine-grained prosodic structure.

Figures (2)

• Figure 1: An example of the proposed VowelPrompt framework for the emotion recognition task.
• Figure 2: Example of a prompt of VowelPrompt combining conversational context, target utterance, and vowel-level prosodic descriptors. The transcript provides lexical content, while each vowel in the target utterance is annotated with interpretable acoustic features, including pitch slope, pitch level and variation, intensity level and variation, and duration. These features are expressed in natural language and integrated into the input to guide the emotion inference by LLM. The example illustrates a frustration-labeled case from IEMOCAP, where prosodic patterns such as high pitch slope and extended vowel duration convey heightened emotional intensity.