Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing AAC Software for Dysarthric Speakers in e-Health Settings: An Evaluation Using TORGO

Macarious Hui, Jinda Zhang, Aanchan Mohan

TL;DR

This work aims to leverage SOTA ASR followed by domain-specific error correction followed by domain-specific error correction to improve ASR for atypical speakers to enable equitable healthcare access both in-person and in e-health settings.

Abstract

Individuals with cerebral palsy (CP) and amyotrophic lateral sclerosis (ALS) frequently face challenges with articulation, leading to dysarthria and resulting in atypical speech patterns. In healthcare settings, communication breakdowns reduce the quality of care. While building an augmentative and alternative communication (AAC) tool to enable fluid communication we found that state-of-the-art (SOTA) automatic speech recognition (ASR) technology like Whisper and Wav2vec2.0 marginalizes atypical speakers largely due to the lack of training data. Our work looks to leverage SOTA ASR followed by domain specific error-correction. English dysarthric ASR performance is often evaluated on the TORGO dataset. Prompt-overlap is a well-known issue with this dataset where phrases overlap between training and test speakers. Our work proposes an algorithm to break this prompt-overlap. After reducing prompt-overlap, results with SOTA ASR models produce extremely high word error rates for speakers with mild and severe dysarthria. Furthermore, to improve ASR, our work looks at the impact of n-gram language models and large-language model (LLM) based multi-modal generative error-correction algorithms like Whispering-LLaMA for a second pass ASR. Our work highlights how much more needs to be done to improve ASR for atypical speakers to enable equitable healthcare access both in-person and in e-health settings.

Enhancing AAC Software for Dysarthric Speakers in e-Health Settings: An Evaluation Using TORGO

TL;DR

This work aims to leverage SOTA ASR followed by domain-specific error correction followed by domain-specific error correction to improve ASR for atypical speakers to enable equitable healthcare access both in-person and in e-health settings.

Abstract

Individuals with cerebral palsy (CP) and amyotrophic lateral sclerosis (ALS) frequently face challenges with articulation, leading to dysarthria and resulting in atypical speech patterns. In healthcare settings, communication breakdowns reduce the quality of care. While building an augmentative and alternative communication (AAC) tool to enable fluid communication we found that state-of-the-art (SOTA) automatic speech recognition (ASR) technology like Whisper and Wav2vec2.0 marginalizes atypical speakers largely due to the lack of training data. Our work looks to leverage SOTA ASR followed by domain specific error-correction. English dysarthric ASR performance is often evaluated on the TORGO dataset. Prompt-overlap is a well-known issue with this dataset where phrases overlap between training and test speakers. Our work proposes an algorithm to break this prompt-overlap. After reducing prompt-overlap, results with SOTA ASR models produce extremely high word error rates for speakers with mild and severe dysarthria. Furthermore, to improve ASR, our work looks at the impact of n-gram language models and large-language model (LLM) based multi-modal generative error-correction algorithms like Whispering-LLaMA for a second pass ASR. Our work highlights how much more needs to be done to improve ASR for atypical speakers to enable equitable healthcare access both in-person and in e-health settings.

Paper Structure

This paper contains 11 sections, 1 equation, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Relation to prior work
  3. The TORGO dataset
  4. Removing prompt overlap in TORGO
  5. Experimental setup
  6. Acoustic models
  7. Language Models
  8. Cross-modal error-correction
  9. Experiments
  10. Discussion
  11. Conclusion

Figures (3)

  • Figure 1: Dysarthric automatic speech recognition followed by error correction
  • Figure 2: Inference samples for error-correction (EC) for speaker M05. Ref shows the reference transcription, ASR shows the transcription output which serves as input to the EC model. Notice how the EC system has memorized transcripts due to prompt overlap in TORGO.
  • Figure 3: Effect of $f$ on the number of utterances retained in training and test set for target speaker M01.