Decoding the Ear: A Framework for Objectifying Expressiveness from Human Preference Through Efficient Alignment

TL;DR

DeEAR introduces a human-aligned, data-efficient metric to quantify speech expressiveness by decomposing it into Emotion, Prosody, and Spontaneity, then fusing sub-dimensions with a learned preference function. The framework achieves strong alignment with human judgments ($PCC=0.91$, $SRCC=0.86$) using under 500 annotations and enables automated SOTA benchmarking ($SRCC=0.96$) and targeted data curation that yields the ExpressiveSpeech dataset (~14K bilingual utterances, 80.2 average expressiveness). An evaluation-driven paradigm demonstrates substantial expressiveness gains in a fine-tuned S2S model ($S_{expr}$ rising from 2.0 to 23.4) and robust cross-domain generalization. This work offers a practical, modular approach for reliable expressiveness assessment and data-driven improvements in expressive speech systems, with potential extensions to reinforcement learning for end-to-end optimization.

Abstract

Recent speech-to-speech (S2S) models generate intelligible speech but still lack natural expressiveness, largely due to the absence of a reliable evaluation metric. Existing approaches, such as subjective MOS ratings, low-level acoustic features, and emotion recognition are costly, limited, or incomplete. To address this, we present DeEAR (Decoding the Expressive Preference of eAR), a framework that converts human preference for speech expressiveness into an objective score. Grounded in phonetics and psychology, DeEAR evaluates speech across three dimensions: Emotion, Prosody, and Spontaneity, achieving strong alignment with human perception (Spearman's Rank Correlation Coefficient, SRCC = 0.86) using fewer than 500 annotated samples. Beyond reliable scoring, DeEAR enables fair benchmarking and targeted data curation. It not only distinguishes expressiveness gaps across S2S models but also selects 14K expressive utterances to form ExpressiveSpeech, which improves the expressive score (from 2.0 to 23.4 on a 100-point scale) of S2S models. Demos and codes are available at https://github.com/FreedomIntelligence/ExpressiveSpeech

Figures (1)

• Figure 1: The DeEAR Framework: (A) The training follows a four-stage pipeline: Stage 1 decomposes expressiveness into Emotion, Prosody, and Spontaneity; Stage 2 trains a scorer for each dimension; Stage 3 learns a Unifier Scorer from these dimension scores; and Stage 4 trains XGBoost to produce the final expressiveness score $S_{\text{expressive}}$. (B) Applications include filtering audio to build high-quality datasets and serving as a reward model to select outputs from generative models. Here, ES denotes the Expressiveness Score.