Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

YOLO-Stutter: End-to-end Region-Wise Speech Dysfluency Detection

Xuanru Zhou, Anshul Kashyap, Steve Li, Ayati Sharma, Brittany Morin, David Baquirin, Jet Vonk, Zoe Ezzes, Zachary Miller, Maria Luisa Gorno Tempini, Jiachen Lian, Gopala Krishna Anumanchipalli

TL;DR

This paper proposes YOLO-Stutter: a first end-to-end method that detects dysfluencies in a time-accurate manner and achieves state-of-the-art performance with a minimum number of trainable parameters for on both simulated data and real aphasia speech.

Abstract

Dysfluent speech detection is the bottleneck for disordered speech analysis and spoken language learning. Current state-of-the-art models are governed by rule-based systems which lack efficiency and robustness, and are sensitive to template design. In this paper, we propose YOLO-Stutter: a first end-to-end method that detects dysfluencies in a time-accurate manner. YOLO-Stutter takes imperfect speech-text alignment as input, followed by a spatial feature aggregator, and a temporal dependency extractor to perform region-wise boundary and class predictions. We also introduce two dysfluency corpus, VCTK-Stutter and VCTK-TTS, that simulate natural spoken dysfluencies including repetition, block, missing, replacement, and prolongation. Our end-to-end method achieves state-of-the-art performance with a minimum number of trainable parameters for on both simulated data and real aphasia speech. Code and datasets are open-sourced at https://github.com/rorizzz/YOLO-Stutter

YOLO-Stutter: End-to-end Region-Wise Speech Dysfluency Detection

TL;DR

This paper proposes YOLO-Stutter: a first end-to-end method that detects dysfluencies in a time-accurate manner and achieves state-of-the-art performance with a minimum number of trainable parameters for on both simulated data and real aphasia speech.

Abstract

Dysfluent speech detection is the bottleneck for disordered speech analysis and spoken language learning. Current state-of-the-art models are governed by rule-based systems which lack efficiency and robustness, and are sensitive to template design. In this paper, we propose YOLO-Stutter: a first end-to-end method that detects dysfluencies in a time-accurate manner. YOLO-Stutter takes imperfect speech-text alignment as input, followed by a spatial feature aggregator, and a temporal dependency extractor to perform region-wise boundary and class predictions. We also introduce two dysfluency corpus, VCTK-Stutter and VCTK-TTS, that simulate natural spoken dysfluencies including repetition, block, missing, replacement, and prolongation. Our end-to-end method achieves state-of-the-art performance with a minimum number of trainable parameters for on both simulated data and real aphasia speech. Code and datasets are open-sourced at https://github.com/rorizzz/YOLO-Stutter
Paper Structure (21 sections, 1 figure, 6 tables)

This paper contains 21 sections, 1 figure, 6 tables.

Table of Contents

  1. Introduction
  2. Naturalistic Dysfluency Simulation
  3. VCTK-Stutter
  4. VCTK-TTS
  5. Method pipeline
  6. TTS rules
  7. Evaluation
  8. End-to-End Dysfluency Detection
  9. Soft speech-text alignments
  10. Spatial feature aggregator
  11. Temporal dependency extractor
  12. YOLO objective
  13. Experiments
  14. Datasets
  15. Training
  16. ...and 6 more sections

Figures (1)

  • Figure 1: YOLO-Stutter method's workflow: starting from reference text and its IPA sequence, by applying TTS-rules and VITS model, we generate the dysfluent speech along with its dysfluent alignment. Utilizing pretrained VITS speech and text encoders, we produce a soft-alignment matrix from the given reference text and mel spectrogram of dysfluent speech. Subsequently, the matrix is then processed through spatial feature aggreator and temporal dependency extractor, leading to predicted targets and bounds. The architecture of our spatial feature aggreator block is illustrated in the left green block. Here are examples of our simulated speech https://bit.ly/3PkKE8W