Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Dysarthric Speech Recognition for Unseen Speakers via Prototype-Based Adaptation

Shiyao Wang, Shiwan Zhao, Jiaming Zhou, Aobo Kong, Yong Qin

TL;DR

This work introduces a prototype-based approach that markedly improves DSR performance for unseen dysarthric speakers without additional fine-tuning, and incorporates supervised contrastive learning to refine feature extraction.

Abstract

Dysarthric speech recognition (DSR) presents a formidable challenge due to inherent inter-speaker variability, leading to severe performance degradation when applying DSR models to new dysarthric speakers. Traditional speaker adaptation methodologies typically involve fine-tuning models for each speaker, but this strategy is cost-prohibitive and inconvenient for disabled users, requiring substantial data collection. To address this issue, we introduce a prototype-based approach that markedly improves DSR performance for unseen dysarthric speakers without additional fine-tuning. Our method employs a feature extractor trained with HuBERT to produce per-word prototypes that encapsulate the characteristics of previously unseen speakers. These prototypes serve as the basis for classification. Additionally, we incorporate supervised contrastive learning to refine feature extraction. By enhancing representation quality, we further improve DSR performance, enabling effective personalized DSR. We release our code at https://github.com/NKU-HLT/PB-DSR.

Enhancing Dysarthric Speech Recognition for Unseen Speakers via Prototype-Based Adaptation

TL;DR

This work introduces a prototype-based approach that markedly improves DSR performance for unseen dysarthric speakers without additional fine-tuning, and incorporates supervised contrastive learning to refine feature extraction.

Abstract

Dysarthric speech recognition (DSR) presents a formidable challenge due to inherent inter-speaker variability, leading to severe performance degradation when applying DSR models to new dysarthric speakers. Traditional speaker adaptation methodologies typically involve fine-tuning models for each speaker, but this strategy is cost-prohibitive and inconvenient for disabled users, requiring substantial data collection. To address this issue, we introduce a prototype-based approach that markedly improves DSR performance for unseen dysarthric speakers without additional fine-tuning. Our method employs a feature extractor trained with HuBERT to produce per-word prototypes that encapsulate the characteristics of previously unseen speakers. These prototypes serve as the basis for classification. Additionally, we incorporate supervised contrastive learning to refine feature extraction. By enhancing representation quality, we further improve DSR performance, enabling effective personalized DSR. We release our code at https://github.com/NKU-HLT/PB-DSR.
Paper Structure (13 sections, 2 equations, 2 figures, 2 tables)

This paper contains 13 sections, 2 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Proposed Methods
  3. Prototype-Based DSR
  4. Combining CTC Loss with SCL Loss
  5. Experiments
  6. Dataset
  7. Experimental Settings
  8. The Settings of DSR Models
  9. The Settings of Prototype-Based DSR
  10. Results and Discussions
  11. Visualization
  12. Conclusions
  13. Acknowledgments

Figures (2)

  • Figure 1: Prototype-based DSR comprises three stages: fine-tuning HuBERT for feature extraction, building per-word prototypes, and prototype-based classification.
  • Figure 2: Visualizations of speech feature distributions: (a) Seen speaker in V. (b) Unseen speaker in R. (c) Unseen speaker in R+. (d) Enhancing R+ by PB-DSR+. Black dots in (d) represent prototypes, while green and red dots denote samples correctly and incorrectly classified by R+, respectively, with their labels from R+ predictions. A red line connects a red dot to its correct prototype by PB-DSR+. Each label is the word ID in the UASpeech dataset.