Application of Whisper in Clinical Practice: the Post-Stroke Speech Assessment during a Naming Task

TL;DR

Post-stroke aphasia assessment is traditionally labor-intensive; this paper evaluates Whisper on a stroke-specific single-word Naming task to determine transcription fidelity and its utility for downstream impairment prediction. By fine-tuning Whisper on age-matched healthy and patient speech, the authors demonstrate substantial improvements in transcription accuracy and show that Whisper-derived representations can support clinically meaningful accuracy-prediction tasks, though generalization to out-of-domain dysarthric speech remains limited. Across datasets, fine-tuning yields large reductions in $WER$ and enables high target-word-detection accuracy, while downstream $F1 Macro$ scores for impairment prediction reach benchmarks suggesting clinical relevance. The results underscore the potential of task- and population-specific adaptation of foundation models to automate post-stroke speech assessment, with clear emphasis on adapting to the disorder and task at hand to ensure robust cross-domain applicability.

Abstract

Detailed assessment of language impairment following stroke remains a cognitively complex and clinician-intensive task, limiting timely and scalable diagnosis. Automatic Speech Recognition (ASR) foundation models offer a promising pathway to augment human evaluation through intelligent systems, but their effectiveness in the context of speech and language impairment remains uncertain. In this study, we evaluate whether Whisper, a state-of-the-art ASR foundation model, can be applied to transcribe and analyze speech from patients with stroke during a commonly used picture-naming task. We assess both verbatim transcription accuracy and the model's ability to support downstream prediction of language function, which has major implications for outcomes after stroke. Our results show that the baseline Whisper model performs poorly on single-word speech utterances. Nevertheless, fine-tuning Whisper significantly improves transcription accuracy (reducing Word Error Rate by 87.72% in healthy speech and 71.22% in speech from patients). Further, learned representations from the model enable accurate prediction of speech quality (average F1 Macro of 0.74 for healthy, 0.75 for patients). However, evaluations on an unseen (TORGO) dataset reveal limited generalizability, highlighting the inability of Whisper to perform zero-shot transcription of single-word utterances on out-of-domain clinical speech and emphasizing the need to adapt models to specific clinical populations. While challenges remain in cross-domain generalization, these findings highlight the potential of foundation models, when appropriately fine-tuned, to advance automated speech and language assessment and rehabilitation for stroke-related impairments.

Figures (2)

• Figure 1: General study pipeline. Overview of the training and inference pipeline for verbatim transcription and accuracy prediction models. Transcription: Training – Whisper models were fine-tuned on speech data from 5 different datasets. A synthetic dataset of audio-transcription word pairs based on stimuli in the Naming task, SONIVA-Naming healthy dataset, collected from healthy age-matched participants performing the Naming task, SONIVA-Naming patient dataset, collected from patients with stroke performing the Naming task, a combined dataset of SONIVA-Naming healthy and patient data, and a combined dataset of all available data (synthetic, SONIVA-Naming healthy and SONIVA-Naming patient). All models were trained using cross-entropy loss to transcribe spoken words (e.g. “acorn”) from the Naming task input audio. Inference – Fine-tuned models (ft-syn, ft-h, ft-p, ft-hp and ft-all) were evaluated on 4 separate datasets – synthetic, SONIVA-Naming healthy, SONIVA-Naming patient, and an unseen testing dataset of dysarthric speech derived from the TORGO database. Accuracy: Training – accuracy prediction models were trained on SONIVA-Naming derived healthy and patient speech in a linear probing framework. The encoder layer of the models was frozen and the classification projection head was trained to predict accuracy scores using cross-entropy loss (e.g “2"). The encoder weights of the models were initialized at baseline or with the weights derived from the models trained for verbatim transcription of healthy or patient speech
• Figure 2: Characteristics of patients identified as impaired (0) or unimpaired (1) from the SONIVA-Naming patient dataset. Predicted impairment status is displayed separately for Semantic (blue), Dysfluency (green), and Phonology (purple) metrics. Speech fluency, previous stroke history, English as second language (other languages), LDL cholesterol level, sex and smoking status were assessed. Error bars indicate standard deviation. Between group significance is denoted with p-values (uncorrected).