Conversational Speech Reveals Structural Robustness Failures in SpeechLLM Backbones

TL;DR

This work evaluates proprietary and open-source LLMs across architectures and scales using the DRES evaluation framework and demonstrates that robustness to speech is shaped by specific training objectives.

Abstract

LLMs serve as the backbone in SpeechLLMs, yet their behavior on spontaneous conversational input remains poorly understood. Conversational speech contains pervasive disfluencies -- interjections, edits, and parentheticals -- that are rare in the written corpora used for pre-training. Because gold disfluency removal is a deletion-only task, it serves as a controlled probe to determine whether a model performs faithful structural repair or biased reinterpretation. Using the DRES evaluation framework, we evaluate proprietary and open-source LLMs across architectures and scales. We show that model performance clusters into stable precision-recall regimes reflecting distinct editing policies. Notably, reasoning models systematically over-delete fluent content, revealing a bias toward semantic abstraction over structural fidelity. While fine-tuning achieves SOTA results, it harms generalization. Our findings demonstrate that robustness to speech is shaped by specific training objectives.

Figures (6)

• Figure 1: Overview of DRES, a factorized structural evaluation framework for SpeechLLM backbones: Gold Switchboard transcripts with annotated disfluencies are provided directly to the LLM backbone to isolate language-level deletion behavior ($D_{\theta}(x)$) from acoustic suppression effects (unobservable $A(w)$) in end-to-end SpeechLLMs. Structural robustness is computed by comparing the model-implied deletion mask to the gold mask, decomposing errors into (i) over-deletion and under-deletion failure modes in the precision–recall space (\ref{['fig:main']}), and (ii) category-specific failure modes (\ref{['fig:clustering']}).
• Figure 2: Precision--recall trade-offs across models under full (top) and segmented (bottom) transcripts. Each point corresponds to a model evaluated at varying in-context learning levels ($k=\{0,1,3,5\}$). Shaded quadrants reveal four editing policies: Under-Deletion ($\mathcal{E}_P \uparrow$, $\mathcal{E}_R \downarrow$) occurs when models fail to recognize conversational structure and leave many disfluencies untouched. Over-Deletion ($\mathcal{E}_P \downarrow$, $\mathcal{E}_R \uparrow$) reflects a rewriting bias: models treat the task as paraphrasing and delete fluent words. Balanced ($\mathcal{E}_P \uparrow$, $\mathcal{E}_R \uparrow$) represents the desired behavior, combining accurate disfluency identification with preservation of fluent content. Poor ($\mathcal{E}_P \downarrow$, $\mathcal{E}_R \downarrow$) inherits the worst of both behaviors, missing disfluencies while also deleting fluent tokens. Proprietary models cluster in the Balanced region, while reasoning models cluster in the Over-Deletion region; small models more frequently occupy the Over-Deletion or Poor regimes. Qualitative structure remains consistent for thresholds in the range $0.55$--$0.70$; quantitative clustering analysis appears in §\ref{['Empirical Evidence of Policy-Level Robustness Patterns']} ($\rhd$ Findings 1, 3, 4, 5).
• Figure 3: Policy structure in robustness space. (a) Clustering in $(\mathcal{E}_P,\mathcal{E}_R)$ space recovers groups that align with the quadrant regimes in Figure \ref{['fig:main']}, indicating that editing policies emerge as geometric structure in robustness space. Black markers denote cluster centers ($\rhd$ Finding 1). (b) Mean category-level $\mathcal{Z}$ scores for each policy cluster ($\mathcal{Z}_E$, $\mathcal{Z}_I$, $\mathcal{Z}_P$) show consistent behavioral differences across disfluency types ($\rhd$ Finding 2).
• Figure 4: Segmentation improves robustness to conversational structure. Segmenting long conversational transcripts consistently improves structural fidelity across models. Performance gains in $\Delta\mathcal{E}_F$ indicate that robustness failures largely arise from long-context instability rather than knowledge limitations. Averaged across $k$ ($\rhd$ Finding 3).
• Figure 5: Scale improves performance, but does not change editing policy. Larger models within each family achieve higher $\mathcal{E}_F$ but reasoning-oriented variants consistently underperform relative to the best performance curve. This shows that scale improves execution of a policy but does not change the underlying editing behavior.($\rhd$ Finding 4)