Lost in Transcription: Identifying and Quantifying the Accuracy Biases of Automatic Speech Recognition Systems Against Disfluent Speech

TL;DR

This work investigates biases in automatic speech recognition against disfluent speech typical of stuttering, using six ASR systems evaluated on real (FluencyBank) and synthetic (LibriSpeech-derived) disfluent datasets. The authors quantify biases with WER, CER, and semantic similarity via $F_{BERT}$, and demonstrate statistically significant degradation in transcription accuracy for disfluent speech. A novel synthetic disfluency generation framework adds controlled disfluency types, enabling robust bias analysis beyond limited real-data samples. The findings highlight the need for inclusive ASR training and evaluation, and point to data collection, human feedback, and targeted mitigation as essential directions for fairer voice technologies. The work introduces a concrete, multi-metric framework for assessing and addressing disfluency-induced biases in ASR.

Abstract

Automatic speech recognition (ASR) systems, increasingly prevalent in education, healthcare, employment, and mobile technology, face significant challenges in inclusivity, particularly for the 80 million-strong global community of people who stutter. These systems often fail to accurately interpret speech patterns deviating from typical fluency, leading to critical usability issues and misinterpretations. This study evaluates six leading ASRs, analyzing their performance on both a real-world dataset of speech samples from individuals who stutter and a synthetic dataset derived from the widely-used LibriSpeech benchmark. The synthetic dataset, uniquely designed to incorporate various stuttering events, enables an in-depth analysis of each ASR's handling of disfluent speech. Our comprehensive assessment includes metrics such as word error rate (WER), character error rate (CER), and semantic accuracy of the transcripts. The results reveal a consistent and statistically significant accuracy bias across all ASRs against disfluent speech, manifesting in significant syntactical and semantic inaccuracies in transcriptions. These findings highlight a critical gap in current ASR technologies, underscoring the need for effective bias mitigation strategies. Addressing this bias is imperative not only to improve the technology's usability for people who stutter but also to ensure their equitable and inclusive participation in the rapidly evolving digital landscape.

Figures (6)

• Figure 1: Transcript produced by an automatic speech recognition (ASR) model from a speech sample of an individual who stutters, sourced from FluencyBank ratner2018fluency. This figure illustrates a real-world scenario where the individual is conducting an interview. Highlighted are both the disfluencies characteristic of stuttered speech and the corresponding transcription inaccuracies produced by the ASR.
• Figure 2: Histogram showing the frequency distribution of utterance durations in seconds for each of the four speech datasets.
• Figure 3: Violin plot illustrating the range and distribution of WER for the wav2vec 2.0 model on FluencyBank (FB) and synthetic (LS) datasets. The plot highlights mean values and variability in WER scores.
• Figure 4: Violin plots illustrating the range and distribution of WER for each ASR model on FluencyBank (FB) and LibriSpeech (LS), both with disfluencies (Y) and without disfluencies (N).
• Figure 5: Violin plots illustrating the range and distribution of CER for each ASR model on FluencyBank (FB) and LibriSpeech (LS), both with disfluencies (Y) and without disfluencies (N).

Theorems & Definitions (2)

• Definition 1
• Definition 2