Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Augmenting Automatic Speech Recognition Models with Disfluency Detection

Robin Amann, Zhaolin Li, Barbara Bruno, Jan Niehues

TL;DR

This work presents an inference-only approach to augment any ASR model with the ability to detect open-set disfluencies, and proposes a modified Connectionist Temporal Classification(CTC)based forced alignment algorithm from [1] to predict wordlevel timestamps while effectively capturing disfluent speech.

Abstract

Speech disfluency commonly occurs in conversational and spontaneous speech. However, standard Automatic Speech Recognition (ASR) models struggle to accurately recognize these disfluencies because they are typically trained on fluent transcripts. Current research mainly focuses on detecting disfluencies within transcripts, overlooking their exact location and duration in the speech. Additionally, previous work often requires model fine-tuning and addresses limited types of disfluencies. In this work, we present an inference-only approach to augment any ASR model with the ability to detect open-set disfluencies. We first demonstrate that ASR models have difficulty transcribing speech disfluencies. Next, this work proposes a modified Connectionist Temporal Classification(CTC)-based forced alignment algorithm from \cite{kurzinger2020ctc} to predict word-level timestamps while effectively capturing disfluent speech. Additionally, we develop a model to classify alignment gaps between timestamps as either containing disfluent speech or silence. This model achieves an accuracy of 81.62% and an F1-score of 80.07%. We test the augmentation pipeline of alignment gap detection and classification on a disfluent dataset. Our results show that we captured 74.13% of the words that were initially missed by the transcription, demonstrating the potential of this pipeline for downstream tasks.

Augmenting Automatic Speech Recognition Models with Disfluency Detection

TL;DR

This work presents an inference-only approach to augment any ASR model with the ability to detect open-set disfluencies, and proposes a modified Connectionist Temporal Classification(CTC)based forced alignment algorithm from [1] to predict wordlevel timestamps while effectively capturing disfluent speech.

Abstract

Speech disfluency commonly occurs in conversational and spontaneous speech. However, standard Automatic Speech Recognition (ASR) models struggle to accurately recognize these disfluencies because they are typically trained on fluent transcripts. Current research mainly focuses on detecting disfluencies within transcripts, overlooking their exact location and duration in the speech. Additionally, previous work often requires model fine-tuning and addresses limited types of disfluencies. In this work, we present an inference-only approach to augment any ASR model with the ability to detect open-set disfluencies. We first demonstrate that ASR models have difficulty transcribing speech disfluencies. Next, this work proposes a modified Connectionist Temporal Classification(CTC)-based forced alignment algorithm from \cite{kurzinger2020ctc} to predict word-level timestamps while effectively capturing disfluent speech. Additionally, we develop a model to classify alignment gaps between timestamps as either containing disfluent speech or silence. This model achieves an accuracy of 81.62% and an F1-score of 80.07%. We test the augmentation pipeline of alignment gap detection and classification on a disfluent dataset. Our results show that we captured 74.13% of the words that were initially missed by the transcription, demonstrating the potential of this pipeline for downstream tasks.
Paper Structure (21 sections, 4 equations, 4 figures, 5 tables)

This paper contains 21 sections, 4 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Disfluency Detection
  3. Augmentation pipeline
  4. Forced alignment
  5. Standard CTC Alignment
  6. Modified CTC Alignment
  7. Cross-attention Alignment
  8. Alignment Comparison Metric
  9. Gap classification
  10. Dataset
  11. Dataset Preparation
  12. Segmenting Audio
  13. Experiments and Results
  14. Experimental setups
  15. Are ASR models good at disfluency recognition?
  16. ...and 6 more sections

Figures (4)

  • Figure 1: The pipeline to augment ASR models with disfluency detection with a follow-up re-transcription for example of application.
  • Figure 2: Alignments of generated transcription to speech signal with standard (upper) and modified (lower) CTC forced alignments. The ASR prediction is: I had that curiosity at the moment, while the manual transcript contains the disfluency: I had that curiosity beside me at the moment.
  • Figure 3: The number of words that are covered by modified alignments with different probability $c$ value. The left is for all words and the right is for the words next to the untranscribed words in the manual transcript.
  • Figure 4: Untranscribed and transcribed words covered by three forced alignment approaches.