Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Measuring the Accuracy of Automatic Speech Recognition Solutions

Korbinian Kuhn, Verena Kersken, Benedikt Reuter, Niklas Egger, Gottfried Zimmermann

TL;DR

Measuring the Accuracy of Automatic Speech Recognition Solutions provides an independent, cross-vendor benchmark of ASR accuracy using a novel Higher Education lecture dataset. The study automates transcription across 11 vendors, applies extensive text normalisation, and evaluates using Word Error Rate (WER) along with alternative metrics, reporting notable variability across vendors, languages, and streaming vs batch modes. Key findings include a roughly 7.0% average WER for English datasets, significantly higher errors for streaming, and only modest gains from adding custom vocabularies, with Whisper (open-source) frequently delivering the best performance. The results highlight reliability gaps in current ASR captioning for accessibility, underscoring the need for improved metrics, diverse datasets, and human oversight to ensure captions meet real-world needs. The work has practical implications for accessibility standards and policy discussions around declaratory rules and quality thresholds for ASR-generated captions.

Abstract

For d/Deaf and hard of hearing (DHH) people, captioning is an essential accessibility tool. Significant developments in artificial intelligence (AI) mean that Automatic Speech Recognition (ASR) is now a part of many popular applications. This makes creating captions easy and broadly available - but transcription needs high levels of accuracy to be accessible. Scientific publications and industry report very low error rates, claiming AI has reached human parity or even outperforms manual transcription. At the same time the DHH community reports serious issues with the accuracy and reliability of ASR. There seems to be a mismatch between technical innovations and the real-life experience for people who depend on transcription. Independent and comprehensive data is needed to capture the state of ASR. We measured the performance of eleven common ASR services with recordings of Higher Education lectures. We evaluated the influence of technical conditions like streaming, the use of vocabularies, and differences between languages. Our results show that accuracy ranges widely between vendors and for the individual audio samples. We also measured a significant lower quality for streaming ASR, which is used for live events. Our study shows that despite the recent improvements of ASR, common services lack reliability in accuracy.

Measuring the Accuracy of Automatic Speech Recognition Solutions

TL;DR

Measuring the Accuracy of Automatic Speech Recognition Solutions provides an independent, cross-vendor benchmark of ASR accuracy using a novel Higher Education lecture dataset. The study automates transcription across 11 vendors, applies extensive text normalisation, and evaluates using Word Error Rate (WER) along with alternative metrics, reporting notable variability across vendors, languages, and streaming vs batch modes. Key findings include a roughly 7.0% average WER for English datasets, significantly higher errors for streaming, and only modest gains from adding custom vocabularies, with Whisper (open-source) frequently delivering the best performance. The results highlight reliability gaps in current ASR captioning for accessibility, underscoring the need for improved metrics, diverse datasets, and human oversight to ensure captions meet real-world needs. The work has practical implications for accessibility standards and policy discussions around declaratory rules and quality thresholds for ASR-generated captions.

Abstract

For d/Deaf and hard of hearing (DHH) people, captioning is an essential accessibility tool. Significant developments in artificial intelligence (AI) mean that Automatic Speech Recognition (ASR) is now a part of many popular applications. This makes creating captions easy and broadly available - but transcription needs high levels of accuracy to be accessible. Scientific publications and industry report very low error rates, claiming AI has reached human parity or even outperforms manual transcription. At the same time the DHH community reports serious issues with the accuracy and reliability of ASR. There seems to be a mismatch between technical innovations and the real-life experience for people who depend on transcription. Independent and comprehensive data is needed to capture the state of ASR. We measured the performance of eleven common ASR services with recordings of Higher Education lectures. We evaluated the influence of technical conditions like streaming, the use of vocabularies, and differences between languages. Our results show that accuracy ranges widely between vendors and for the individual audio samples. We also measured a significant lower quality for streaming ASR, which is used for live events. Our study shows that despite the recent improvements of ASR, common services lack reliability in accuracy.
Paper Structure (31 sections, 6 figures, 4 tables)

This paper contains 31 sections, 6 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Transcription with ASR
  4. Benchmarking ASR
  5. Measuring transcription accuracy
  6. Comparison of ASR services
  7. Methodology
  8. Dataset
  9. Recording scenario
  10. Selection criteria
  11. Bias
  12. Data preparation
  13. Availability
  14. Additional variables
  15. Transcription
  16. ...and 16 more sections

Figures (6)

  • Figure 1: Methodology
  • Figure 2: Word Error Rate by vendor for the English datasets
  • Figure 3: Word Error Rate by file for the English datasets
  • Figure 4: Average hits of vocabulary words by vendor for the English datasets
  • Figure 5: Average Word Error Rate in percent to confidence in percent by vendor for all datasets
  • ...and 1 more figures