Speaker-Independent Dysarthria Severity Classification using Self-Supervised Transformers and Multi-Task Learning

TL;DR

A novel machine-learning framework using transformers for stratifying and monitoring patient speech and demonstrating robustness and generalisability for speaker-independent dysarthria severity classification, with potential implications for broader clinical applications in automated dysarthria assessments.

Abstract

Dysarthria, a condition resulting from impaired control of the speech muscles due to neurological disorders, significantly impacts the communication and quality of life of patients. The condition's complexity, human scoring and varied presentations make its assessment and management challenging. This study presents a transformer-based framework for automatically assessing dysarthria severity from raw speech data. It can offer an objective, repeatable, accessible, standardised and cost-effective and compared to traditional methods requiring human expert assessors. We develop a transformer framework, called Speaker-Agnostic Latent Regularisation (SALR), incorporating a multi-task learning objective and contrastive learning for speaker-independent multi-class dysarthria severity classification. The multi-task framework is designed to reduce reliance on speaker-specific characteristics and address the intrinsic intra-class variability of dysarthric speech. We evaluated on the Universal Access Speech dataset using leave-one-speaker-out cross-validation, our model demonstrated superior performance over traditional machine learning approaches, with an accuracy of $70.48\%$ and an F1 score of $59.23\%$. Our SALR model also exceeded the previous benchmark for AI-based classification, which used support vector machines, by $16.58\%$. We open the black box of our model by visualising the latent space where we can observe how the model substantially reduces speaker-specific cues and amplifies task-specific ones, thereby showing its robustness. In conclusion, SALR establishes a new benchmark in speaker-independent multi-class dysarthria severity classification using generative AI. The potential implications of our findings for broader clinical applications in automated dysarthria severity assessments.

Figures (6)

• Figure 1: Architecture A. Overview of the Speaker-Agnostic Latent Regularisation (SALR) Framework. B. Detailed Visualisation of the Architecture and Training Procedure of the Speaker-Agnostic Latent Regularisation (SALR) Framework.
• Figure 2: Normalised confusion matrices of A. fine-tuned Wav2Vec 2.0 model, B. SALR framework
• Figure 3: Visualisation of t-SNE embeddings from different models with data points coloured according to patient severity (first row) and patient code (second row). A, C depict the fine-tuned Wav2Vec 2.0 model, B, D showcase the SALR framework. These visualisations support our hypothesis that the SALR framework organises the latent space in alignment with severity levels (as seen in the first row) and disperses speaker clusters (as seen in the second row).
• Figure 4: Comparative performance in speaker-independent multi-class dysarthria severity classification on the UA-Speech dataset(uncommon words). The bar chart illustrates the performance of our models in comparison to the existing benchmark. tripathi2020improved.
• Figure :