A Multilingual Framework for Dysarthria: Detection, Severity Classification, Speech-to-Text, and Clean Speech Generation
Ananya Raghu, Anisha Raghu, Nithika Vivek, Sofie Budman, Omar Mansour
TL;DR
This work introduces a multilingual dysarthria framework that unifies detection, severity assessment, clean-speech generation, speech-to-text, emotion detection, and voice cloning across English, Russian, and German. It combines MFCC-based CNNs for detection, spectrogram CNNs for severity with interpretable Grad-CAM, a two-stage U-Net translation pipeline for cross-lingual clean speech, and a Whisper Tiny–LLASA-3B driven voice-cloning setup, achieving $97.5\%$ detection accuracy and $97.64\%$ severity accuracy, with cross-language transfer demonstrating promise for low-resource languages. The system also demonstrates practical end-to-end capabilities, including $WER=0.1367$ after three epochs and effective emotion classification, enabling richer patient communication. The results highlight the potential for robust, multilingual dysarthria assessment and assistive technologies, paving the way for globally inclusive diagnostic tools and communication aids.
Abstract
Dysarthria is a motor speech disorder that results in slow and often incomprehensible speech. Speech intelligibility significantly impacts communication, leading to barriers in social interactions. Dysarthria is often a characteristic of neurological diseases including Parkinson's and ALS, yet current tools lack generalizability across languages and levels of severity. In this study, we present a unified AI-based multilingual framework that addresses six key components: (1) binary dysarthria detection, (2) severity classification, (3) clean speech generation, (4) speech-to-text conversion, (5) emotion detection, and (6) voice cloning. We analyze datasets in English, Russian, and German, using spectrogram-based visualizations and acoustic feature extraction to inform model training. Our binary detection model achieved 97% accuracy across all three languages, demonstrating strong generalization across languages. The severity classification model also reached 97% test accuracy, with interpretable results showing model attention focused on lower harmonics. Our translation pipeline, trained on paired Russian dysarthric and clean speech, reconstructed intelligible outputs with low training (0.03) and test (0.06) L1 losses. Given the limited availability of English dysarthric-clean pairs, we fine-tuned the Russian model on English data and achieved improved losses of 0.02 (train) and 0.03 (test), highlighting the promise of cross-lingual transfer learning for low-resource settings. Our speech-to-text pipeline achieved a Word Error Rate of 0.1367 after three epochs, indicating accurate transcription on dysarthric speech and enabling downstream emotion recognition and voice cloning from transcribed speech. Overall, the results and products of this study can be used to diagnose dysarthria and improve communication and understanding for patients across different languages.