Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VocalBench-DF: A Benchmark for Evaluating Speech LLM Robustness to Disfluency

Hongcheng Liu, Yixuan Hou, Heyang Liu, Yuhao Wang, Yanfeng Wang, Yu Wang

TL;DR

VocalBench-DF introduces a structured benchmark to evaluate Speech-LLMs under diverse disfluencies, emphasizing speech-signal properties like phoneme structure and prosody. By cataloging disfluencies along linguistic realization and interaction interference axes and testing 22 mainstream Speech-LLMs, the study reveals substantial brittleness, with phoneme-level disfluencies and long-context challenges as key bottlenecks. The analysis demonstrates the complementary benefits of strong speech recognition and advanced reasoning, suggesting that robust disfluency handling requires coordinated component and training improvements rather than scaling alone. The work highlights a pressing need for methods that improve disfluency processing to build truly inclusive Speech-LLMs applicable to users with speech impairments. This benchmark and its findings lay a foundation for targeted robustness enhancements in end-to-end speech-based AI systems.

Abstract

While Speech Large Language Models (Speech-LLMs) show strong performance in many applications, their robustness is critically under-tested, especially to speech disfluency. Existing evaluations often rely on idealized inputs, overlooking common disfluencies, particularly those associated with conditions like Parkinson's disease. This work investigates whether current Speech-LLMs can maintain performance when interacting with users who have speech impairments. To facilitate this inquiry, we introduce VocalBench-DF, a framework for the systematic evaluation of disfluency across a multi-dimensional taxonomy. Our evaluation of 22 mainstream Speech-LLMs reveals substantial performance degradation, indicating that their real-world readiness is limited. Further analysis identifies phoneme-level processing and long-context modeling as primary bottlenecks responsible for these failures. Strengthening recognition and reasoning capability from components and pipelines can substantially improve robustness. These findings highlight the urgent need for new methods to improve disfluency handling and build truly inclusive Speech-LLMs

VocalBench-DF: A Benchmark for Evaluating Speech LLM Robustness to Disfluency

TL;DR

VocalBench-DF introduces a structured benchmark to evaluate Speech-LLMs under diverse disfluencies, emphasizing speech-signal properties like phoneme structure and prosody. By cataloging disfluencies along linguistic realization and interaction interference axes and testing 22 mainstream Speech-LLMs, the study reveals substantial brittleness, with phoneme-level disfluencies and long-context challenges as key bottlenecks. The analysis demonstrates the complementary benefits of strong speech recognition and advanced reasoning, suggesting that robust disfluency handling requires coordinated component and training improvements rather than scaling alone. The work highlights a pressing need for methods that improve disfluency processing to build truly inclusive Speech-LLMs applicable to users with speech impairments. This benchmark and its findings lay a foundation for targeted robustness enhancements in end-to-end speech-based AI systems.

Abstract

While Speech Large Language Models (Speech-LLMs) show strong performance in many applications, their robustness is critically under-tested, especially to speech disfluency. Existing evaluations often rely on idealized inputs, overlooking common disfluencies, particularly those associated with conditions like Parkinson's disease. This work investigates whether current Speech-LLMs can maintain performance when interacting with users who have speech impairments. To facilitate this inquiry, we introduce VocalBench-DF, a framework for the systematic evaluation of disfluency across a multi-dimensional taxonomy. Our evaluation of 22 mainstream Speech-LLMs reveals substantial performance degradation, indicating that their real-world readiness is limited. Further analysis identifies phoneme-level processing and long-context modeling as primary bottlenecks responsible for these failures. Strengthening recognition and reasoning capability from components and pipelines can substantially improve robustness. These findings highlight the urgent need for new methods to improve disfluency handling and build truly inclusive Speech-LLMs
Paper Structure (35 sections, 4 figures, 21 tables)

This paper contains 35 sections, 4 figures, 21 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Benchmark Construction
  4. Linguistic Realization Disfluency
  5. Content Insertion
  6. Repetition
  7. Filler Words
  8. Speech Pause
  9. Utterance Modification
  10. Reback
  11. Restart
  12. Correction
  13. Interaction Interference Disfluency
  14. White Noise
  15. Background Music
  16. ...and 20 more sections

Figures (4)

  • Figure 1: Illustration of the current speech-LLM's various performance under fluent and disfluent speech. The bottom part presents the GLM-4-Voice's performance under various conditions, where almost all disfluent types exhibit huge degradation.
  • Figure 2: The demonstration of our dataset. The left part is the details about two categories and nine sub-tasks, and the right part is the whole pipeline of data construction and evaluation.
  • Figure 3: Average performance of Speech-LLMs across three disfluency severity levels. There are consistent stepwise declines for most types. (W-Rep: word repetition, HP-Rep: head phoneme repetition, AP-Rep: any phoneme repetition, Comb-Rep: combined repetition)
  • Figure 4: Performance of combinations of ASR models and LLMs. Each point denotes a score in a specific disfluency type. Whiskers show the range of non-outlier data. The box shows the interquartile range from $25\%$ to $75\%$. The line inside the box marks the median.