Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Self-supervised Speech Models for Word-Level Stuttered Speech Detection

Yi-Jen Shih, Zoi Gkalitsiou, Alexandros G. Dimakis, David Harwath

TL;DR

This paper tackles the inadequacy of utterance-level stuttering detection for clinical use by introducing word-level stuttering detection via self-supervised speech models. It employs a WavLM Large backbone with a Hierarchical Convolution interface and an auxiliary CTC objective, trained in two stages: LibriSpeech-based pretraining with synthetic disfluencies and SEP-28K fine-tuning on clinically annotated data aligned to word-level labels. The authors curate a word-level stuttering dataset, demonstrate state-of-the-art performance over prior methods on both utterance- and word-level tasks, and provide extensive ablations to illuminate the impact of the SSL backbone, CTC loss, and training data size. The work advances clinically relevant, edge-deployable screening capabilities and highlights future directions in data diversity and word-level stuttering type classification.

Abstract

Clinical diagnosis of stuttering requires an assessment by a licensed speech-language pathologist. However, this process is time-consuming and requires clinicians with training and experience in stuttering and fluency disorders. Unfortunately, only a small percentage of speech-language pathologists report being comfortable working with individuals who stutter, which is inadequate to accommodate for the 80 million individuals who stutter worldwide. Developing machine learning models for detecting stuttered speech would enable universal and automated screening for stuttering, enabling speech pathologists to identify and follow up with patients who are most likely to be diagnosed with a stuttering speech disorder. Previous research in this area has predominantly focused on utterance-level detection, which is not sufficient for clinical settings where word-level annotation of stuttering is the norm. In this study, we curated a stuttered speech dataset with word-level annotations and introduced a word-level stuttering speech detection model leveraging self-supervised speech models. Our evaluation demonstrates that our model surpasses previous approaches in word-level stuttering speech detection. Additionally, we conducted an extensive ablation analysis of our method, providing insight into the most important aspects of adapting self-supervised speech models for stuttered speech detection.

Self-supervised Speech Models for Word-Level Stuttered Speech Detection

TL;DR

This paper tackles the inadequacy of utterance-level stuttering detection for clinical use by introducing word-level stuttering detection via self-supervised speech models. It employs a WavLM Large backbone with a Hierarchical Convolution interface and an auxiliary CTC objective, trained in two stages: LibriSpeech-based pretraining with synthetic disfluencies and SEP-28K fine-tuning on clinically annotated data aligned to word-level labels. The authors curate a word-level stuttering dataset, demonstrate state-of-the-art performance over prior methods on both utterance- and word-level tasks, and provide extensive ablations to illuminate the impact of the SSL backbone, CTC loss, and training data size. The work advances clinically relevant, edge-deployable screening capabilities and highlights future directions in data diversity and word-level stuttering type classification.

Abstract

Clinical diagnosis of stuttering requires an assessment by a licensed speech-language pathologist. However, this process is time-consuming and requires clinicians with training and experience in stuttering and fluency disorders. Unfortunately, only a small percentage of speech-language pathologists report being comfortable working with individuals who stutter, which is inadequate to accommodate for the 80 million individuals who stutter worldwide. Developing machine learning models for detecting stuttered speech would enable universal and automated screening for stuttering, enabling speech pathologists to identify and follow up with patients who are most likely to be diagnosed with a stuttering speech disorder. Previous research in this area has predominantly focused on utterance-level detection, which is not sufficient for clinical settings where word-level annotation of stuttering is the norm. In this study, we curated a stuttered speech dataset with word-level annotations and introduced a word-level stuttering speech detection model leveraging self-supervised speech models. Our evaluation demonstrates that our model surpasses previous approaches in word-level stuttering speech detection. Additionally, we conducted an extensive ablation analysis of our method, providing insight into the most important aspects of adapting self-supervised speech models for stuttered speech detection.
Paper Structure (15 sections, 4 equations, 3 figures, 3 tables)

This paper contains 15 sections, 4 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Model Architecture
  5. Implementation Details
  6. Word Level Stuttering Speech Dataset
  7. Evaluation
  8. Utterance Level stuttered speech detection
  9. Word Level stuttered speech detection
  10. The effect of CTC loss and SSL interface selection
  11. The effect of different Speech SSL models
  12. The effect of fine-tuning dataset size
  13. Qualitative results of Stuttering Detection
  14. Conclusions
  15. Acknowledgement

Figures (3)

  • Figure 1: Overall framework diagram. The transformer layers and Convolution Layers are initialized from off-the-shelf WavLM Large and remain frozen during pretrain and finetune stage. "Hierarchical Conv." stands for "Hierarchical Convolution".
  • Figure 2: The learned weights distribution of weighted sum interface in WavLMLg + WS and WavLMLg + WS + CTC.
  • Figure 3: Examples of our model and baseline model on word-level stuttering speech detection. Only the words with underscores are considered when evaluating. Words in red indicate that it is categorized as stuttered speech.