The Sound of Syntax: Finetuning and Comprehensive Evaluation of Language Models for Speech Pathology

TL;DR

This work addresses the gap in clinically robust AI support for pediatric speech-language pathology by building a taxonomy of real-world MLM use cases and delivering the first comprehensive benchmark across five clinical tasks with 1,000 annotated data points per task. It evaluates 15 state-of-the-art MLMs, comparing audio-grounded and transcription-based pipelines, and demonstrates that no model yet meets clinical reliability, though targeted domain fine-tuning yields meaningful improvements. The study also provides extensive analyses of robustness, biases (notably gender and language effects), and the impact of reasoning strategies, ensemble methods, and background noise, offering practical guidance for deploying AI in SLP settings. Overall, the results highlight both the potential and current limitations of MLMs in pediatric SLP, emphasizing the need for careful task-specific adaptation, fairness considerations, and privacy-preserving approaches for real-world use. The work advances reproducible benchmarking in SLP and lays out clear directions for future research and responsible clinical validation.

Abstract

According to the U.S. National Institutes of Health, more than 3.4 million children experience speech disorders that require clinical intervention. The number of speech-language pathologists (SLPs) is roughly 20 times fewer than the number of affected children, highlighting a significant gap in children's care and a pressing need for technological support that improves the productivity of SLPs. State-of-the-art multimodal language models (MLMs) show promise for supporting SLPs, but their use remains underexplored largely due to a limited understanding of their performance in high-stakes clinical settings. To address this gap, we collaborate with domain experts to develop a taxonomy of real-world use cases of MLMs in speech-language pathologies. Building on this taxonomy, we introduce the first comprehensive benchmark for evaluating MLM across five core use cases, each containing 1,000 manually annotated data points. This benchmark includes robustness and sensitivity tests under various settings, including background noise, speaker gender, and accent. Our evaluation of 15 state-of-the-art MLMs reveals that no single model consistently outperforms others across all tasks. Notably, we find systematic disparities, with models performing better on male speakers, and observe that chain-of-thought prompting can degrade performance on classification tasks with large label spaces and narrow decision boundaries. Furthermore, we study fine-tuning MLMs on domain-specific data, achieving improvements of over 10\% compared to base models. These findings highlight both the potential and limitations of current MLMs for speech-language pathology applications, underscoring the need for further research and targeted development.

Figures (8)

• Figure 1: Taxonomy of classification tasks in SLPHelm. The benchmark includes three core diagnostic tasks: (i) disorder diagnosis, (ii) disorder symptom classification, and (iii) disorder type classification.
• Figure 2: Metrics across all scenarios
• Figure 3: Model performance after finetuning
• Figure 4: Model performance across robustness under noisy conditions, across gender, and languages
• Figure 5: Model performance across age groups