Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Lost in Phonation: Voice Quality Variation as an Evaluation Dimension for Speech Foundation Models

Harm Lameris, Shree Harsha Bokkahalli Satish, Joakim Gustafson, Éva Székely

TL;DR

The paper investigates how voice-quality variation—specifically creaky and breathy phonation—affects speech foundation models (SFMs) beyond standard MCQA benchmarks. It introduces VQ-Bench, a parallel-synthesis evaluation suite that tests open-ended generation and speech emotion recognition under controlled phonation changes using Buckeye and VCTK references and zero-shot synthesis with F5-TTS. The study finds that phonation type meaningfully shifts model outputs and emotion predictions, aligning with known human perceptual biases, and highlights potential gender-related biases and deployment concerns. This work provides a reproducible protocol for probing paralinguistic variation in SFMs and emphasizes the importance of accounting for voice quality in responsible AI deployment.

Abstract

Recent advances in speech foundation models (SFMs) have enabled the direct processing of spoken language from raw audio, bypassing intermediate textual representations. This capability allows SFMs to be exposed to, and potentially respond to, rich paralinguistic variations embedded in the input speech signal. One under-explored dimension of paralinguistic variation is voice quality, encompassing phonation types such as creaky and breathy voice. These phonation types are known to influence how listeners infer affective state, stance and social meaning in speech. Existing benchmarks for speech understanding largely rely on multiple-choice question answering (MCQA) formats, which are prone to failure and therefore unreliable in capturing the nuanced ways paralinguistic features influence model behaviour. In this paper, we probe SFMs through open-ended generation tasks and speech emotion recognition, evaluating whether model behaviours are consistent across different phonation inputs. We introduce a new parallel dataset featuring synthesized modifications to voice quality, designed to evaluate SFM responses to creaky and breathy voice. Our work provides the first examination of SFM sensitivity to these particular non-lexical aspects of speech perception.

Lost in Phonation: Voice Quality Variation as an Evaluation Dimension for Speech Foundation Models

TL;DR

The paper investigates how voice-quality variation—specifically creaky and breathy phonation—affects speech foundation models (SFMs) beyond standard MCQA benchmarks. It introduces VQ-Bench, a parallel-synthesis evaluation suite that tests open-ended generation and speech emotion recognition under controlled phonation changes using Buckeye and VCTK references and zero-shot synthesis with F5-TTS. The study finds that phonation type meaningfully shifts model outputs and emotion predictions, aligning with known human perceptual biases, and highlights potential gender-related biases and deployment concerns. This work provides a reproducible protocol for probing paralinguistic variation in SFMs and emphasizes the importance of accounting for voice quality in responsible AI deployment.

Abstract

Recent advances in speech foundation models (SFMs) have enabled the direct processing of spoken language from raw audio, bypassing intermediate textual representations. This capability allows SFMs to be exposed to, and potentially respond to, rich paralinguistic variations embedded in the input speech signal. One under-explored dimension of paralinguistic variation is voice quality, encompassing phonation types such as creaky and breathy voice. These phonation types are known to influence how listeners infer affective state, stance and social meaning in speech. Existing benchmarks for speech understanding largely rely on multiple-choice question answering (MCQA) formats, which are prone to failure and therefore unreliable in capturing the nuanced ways paralinguistic features influence model behaviour. In this paper, we probe SFMs through open-ended generation tasks and speech emotion recognition, evaluating whether model behaviours are consistent across different phonation inputs. We introduce a new parallel dataset featuring synthesized modifications to voice quality, designed to evaluate SFM responses to creaky and breathy voice. Our work provides the first examination of SFM sensitivity to these particular non-lexical aspects of speech perception.

Paper Structure

This paper contains 14 sections, 1 equation, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Method
  3. VQ-Bench Creation
  4. Long-form task creation
  5. Target voice synthesis
  6. Voice quality conversion
  7. Experiments
  8. Results
  9. Discussion
  10. Conclusion
  11. Bibliographical References
  12. Language Resource References
  13. Appendix: Model Bias Evaluation Dimensions
  14. Appendix: Speech Emotion Recognition Results

Figures (3)

  • Figure 1: The H1--H2 values for the synthesized prompts for each voice quality by gender.
  • Figure 2: The H1--A3 values for the synthesized prompts for each voice quality by gender.
  • Figure 3: Long-form evaluation prompt variations across four real-world usage categories. Text prompts remain the same as in bokkahallisatish2025biasbenchmarks