Evaluating Speech-to-Text Systems with PennSound

TL;DR

This workbenchmarks multiple commercial and open-source speech-to-text systems on PennSound, the world's largest online poetry audio collection, using a random sample of 100 complete recordings. It assesses word recognition with Word Error Rate ($WER$) via SCTK and diarization with Diarization Error Rate ($DER$) via dscore, while also examining Whisper hallucinations and the impact of runtime options. Rev.ai achieves the best $WER$ overall, Whisper performs best among open-source options when hallucinations are controlled, and AWS leads in $DER$, with overall differences being modest and tradeoffs depending on user priorities. The study provides a practical benchmark and data resources to support text-based indexing and cross-system comparisons for large, diverse audio corpora, and highlights the need to manage hallucinations in open-source models for reliable deployment.

Abstract

A random sample of nearly 10 hours of speech from PennSound, the world's largest online collection of poetry readings and discussions, was used as a benchmark to evaluate several commercial and open-source speech-to-text systems. PennSound's wide variation in recording conditions and speech styles makes it a good representative for many other untranscribed audio collections. Reference transcripts were created by trained annotators, and system transcripts were produced from AWS, Azure, Google, IBM, NeMo, Rev.ai, Whisper, and Whisper.cpp. Based on word error rate, Rev.ai was the top performer, and Whisper was the top open source performer (as long as hallucinations were avoided). AWS had the best diarization error rates among three systems. However, WER and DER differences were slim, and various tradeoffs may motivate choosing different systems for different end users. We also examine the issue of hallucinations in Whisper. Users of Whisper should be cautioned to be aware of runtime options, and whether the speed vs accuracy trade off is acceptable.

Figures (5)

• Figure 1: Distribution of full length recording durations.
• Figure 2: WERs, Sorted Separately
• Figure 3: DERs, Sorted Separately
• Figure 4: IBM SNR
• Figure 5: Unintelligible regions and overlapping speech explain some high WERs