Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Multi-modal Approach to Dysarthria Detection and Severity Assessment Using Speech and Text Information

Anuprabha M, Krishna Gurugubelli, V Kesavaraj, Anil Kumar Vuppala

TL;DR

The paper tackles dysarthria detection and severity assessment by introducing a multi-modal framework that fuses speech and text using a cross-attention mechanism to learn acoustic-linguistic correspondences. It formalizes the problem as predicting $C$ from $S$ and $T$ via $P(C|S,T)$ and demonstrates a three-component architecture: a speech encoder, a text encoder, and a cross-modal classifier that jointly produce detection and severity outputs. Empirical results on the UA-Speech dataset show that the speech-text model outperforms speech-only baselines across most settings, with notable gains in speaker-dependent and unseen-word scenarios, and nuanced improvements across word groups. The method achieves practical impact by providing more robust, linguistically informed dysarthria diagnostics, with higher detection accuracy (up to 99.53% in SD) and improved severity assessment, particularly in SID settings; future work will optimize fusion strategies for even more reliable clinical deployment.

Abstract

Automatic detection and severity assessment of dysarthria are crucial for delivering targeted therapeutic interventions to patients. While most existing research focuses primarily on speech modality, this study introduces a novel approach that leverages both speech and text modalities. By employing cross-attention mechanism, our method learns the acoustic and linguistic similarities between speech and text representations. This approach assesses specifically the pronunciation deviations across different severity levels, thereby enhancing the accuracy of dysarthric detection and severity assessment. All the experiments have been performed using UA-Speech dysarthric database. Improved accuracies of 99.53% and 93.20% in detection, and 98.12% and 51.97% for severity assessment have been achieved when speaker-dependent and speaker-independent, unseen and seen words settings are used. These findings suggest that by integrating text information, which provides a reference linguistic knowledge, a more robust framework has been developed for dysarthric detection and assessment, thereby potentially leading to more effective diagnoses.

A Multi-modal Approach to Dysarthria Detection and Severity Assessment Using Speech and Text Information

TL;DR

The paper tackles dysarthria detection and severity assessment by introducing a multi-modal framework that fuses speech and text using a cross-attention mechanism to learn acoustic-linguistic correspondences. It formalizes the problem as predicting from and via and demonstrates a three-component architecture: a speech encoder, a text encoder, and a cross-modal classifier that jointly produce detection and severity outputs. Empirical results on the UA-Speech dataset show that the speech-text model outperforms speech-only baselines across most settings, with notable gains in speaker-dependent and unseen-word scenarios, and nuanced improvements across word groups. The method achieves practical impact by providing more robust, linguistically informed dysarthria diagnostics, with higher detection accuracy (up to 99.53% in SD) and improved severity assessment, particularly in SID settings; future work will optimize fusion strategies for even more reliable clinical deployment.

Abstract

Automatic detection and severity assessment of dysarthria are crucial for delivering targeted therapeutic interventions to patients. While most existing research focuses primarily on speech modality, this study introduces a novel approach that leverages both speech and text modalities. By employing cross-attention mechanism, our method learns the acoustic and linguistic similarities between speech and text representations. This approach assesses specifically the pronunciation deviations across different severity levels, thereby enhancing the accuracy of dysarthric detection and severity assessment. All the experiments have been performed using UA-Speech dysarthric database. Improved accuracies of 99.53% and 93.20% in detection, and 98.12% and 51.97% for severity assessment have been achieved when speaker-dependent and speaker-independent, unseen and seen words settings are used. These findings suggest that by integrating text information, which provides a reference linguistic knowledge, a more robust framework has been developed for dysarthric detection and assessment, thereby potentially leading to more effective diagnoses.

Paper Structure

This paper contains 16 sections, 3 equations, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Proposed approach
  3. Problem Formulation
  4. Proposed Architecture
  5. Speech Encoder
  6. Text Encoder
  7. Multi-modal Classifier
  8. Experimental setup
  9. Database
  10. Comprehensive Details of Different Settings
  11. Implementation Details
  12. Results and Discussion
  13. Comparative Analysis of Speech-Only and Speech-Text Modalities
  14. Performance Assessment of Proposed Framework Across Diverse Word Groups
  15. Comparison with State-of-the-Art Systems
  16. ...and 1 more sections

Figures (1)

  • Figure 1: Proposed architecture for detection and severity assessment of dysarthria