Towards Robust Dysarthric Speech Recognition: LLM-Agent Post-ASR Correction Beyond WER

TL;DR

This work tackles the misalignment between WER and semantic fidelity in dysarthric speech recognition by introducing a Judge-Editor Agent (JEA) that post-corrects ASR outputs across top-$k$ hypotheses to preserve meaning. It releases SAP-Hypo5, a 35k-utterance dysarthric benchmark with reference transcripts and ASR hypotheses, enabling fine-grained, multi-metric evaluation including semantic and SLU tasks. Through zero-shot prompting and lightweight LoRA-based fine-tuning of several LLMs, the approach achieves a 14.51% WER reduction and notable semantic gains, with fine-tuned models outperforming zero-shot in most metrics. The results highlight that semantic and downstream task metrics correlate better with real-world utility than WER under domain shift, suggesting a shift toward semantic-focused evaluation and training for robust dysarthric ASR systems.

Abstract

While Automatic Speech Recognition (ASR) is typically benchmarked by word error rate (WER), real-world applications ultimately hinge on semantic fidelity. This mismatch is particularly problematic for dysarthric speech, where articulatory imprecision and disfluencies can cause severe semantic distortions. To bridge this gap, we introduce a Large Language Model (LLM)-based agent for post-ASR correction: a Judge-Editor over the top-k ASR hypotheses that keeps high-confidence spans, rewrites uncertain segments, and operates in both zero-shot and fine-tuned modes. In parallel, we release SAP-Hypo5, the largest benchmark for dysarthric speech correction, to enable reproducibility and future exploration. Under multi-perspective evaluation, our agent achieves a 14.51% WER reduction alongside substantial semantic gains, including a +7.59 pp improvement in MENLI and +7.66 pp in Slot Micro F1 on challenging samples. Our analysis further reveals that WER is highly sensitive to domain shift, whereas semantic metrics correlate more closely with downstream task performance.

Figures (1)

• Figure 1: Judge–Editor Agent (JEA). Given the ASR multiple hypotheses, JEA first judges span-level uncertainty and cross-hypothesis consistency, then editing and fusion spans to synthesize a single intent-preserving transcript.