Voice Quality Dimensions as Interpretable Primitives for Speaking Style for Atypical Speech and Affect

TL;DR

The paper addresses how to interpret speaking style in atypical speech and affect by learning seven interpretable voice quality dimensions (VQDs) from frozen audio embeddings. It trains linear probes on features from multiple pre-trained models using the Speech Accessibility Project (SAP) data to predict VQD ratings and evaluates generalization across speech categories, unseen datasets, and languages, including Italian and affective data. The results show strong predictive performance and robust zero-shot transfer, with HuBERT ASR features excelling for intelligibility/imprecise consonants and CLAP for harshness; SpICE baselines are outperformed in out-of-domain settings. These findings support using VQDs as interpretable primitives to improve accessibility and robustness in speaking-style tasks, while outlining future work on adapters and broader label mappings to deepen interpretability.

Abstract

Perceptual voice quality dimensions describe key characteristics of atypical speech and other speech modulations. Here we develop and evaluate voice quality models for seven voice and speech dimensions (intelligibility, imprecise consonants, harsh voice, naturalness, monoloudness, monopitch, and breathiness). Probes were trained on the public Speech Accessibility (SAP) project dataset with 11,184 samples from 434 speakers, using embeddings from frozen pre-trained models as features. We found that our probes had both strong performance and strong generalization across speech elicitation categories in the SAP dataset. We further validated zero-shot performance on additional datasets, encompassing unseen languages and tasks: Italian atypical speech, English atypical speech, and affective speech. The strong zero-shot performance and the interpretability of results across an array of evaluations suggests the utility of using voice quality dimensions in speaking style-related tasks.

Figures (6)

• Figure 1: Pearson correlations between annotations
• Figure 2: Distributions of annotations for imprecise consonants and naturalness for each speech category
• Figure 3: Regression probes, showing clear progression of predicted scores with rated severity
• Figure 4: Zero-shot predictions from the HuBERT probe for each voice quality dimension on the Eds-vc dataset and on the EasyCall dataset, stratified by rated speech severity
• Figure 5: Zero-shot predictions for each voice quality dimension on RAVDESS using probes trained on HuBERT for each categorical emotion