Earnings-22: A Practical Benchmark for Accents in the Wild

TL;DR

Earnings-22 provides a real-world accented English benchmark with 119 hours of earnings-call audio across seven regional groups to probe ASR performance. The study benchmarks four commercial providers, conducts region- and word-level analyses, and uses Monte Carlo permutation tests to demonstrate statistically significant accent biases. It further analyzes transcript features such as filled pauses and word fragments to understand their differential impact by region, revealing that accent-bias persists despite overall WER improvements. The dataset aims to bridge academia and industry by offering a freely available, domain-specific benchmark to drive improvements in robust, equitable ASR for diverse voices.

Abstract

Modern automatic speech recognition (ASR) systems have achieved superhuman Word Error Rate (WER) on many common corpora despite lacking adequate performance on speech in the wild. Beyond that, there is a lack of real-world, accented corpora to properly benchmark academic and commercial models. To ensure this type of speech is represented in ASR benchmarking, we present Earnings-22, a 125 file, 119 hour corpus of English-language earnings calls gathered from global companies. We run a comparison across 4 commercial models showing the variation in performance when taking country of origin into consideration. Looking at hypothesis transcriptions, we explore errors common to all ASR systems tested. By examining Individual Word Error Rate (IWER), we find that key speech features impact model performance more for certain accents than others. Earnings-22 provides a free-to-use benchmark of real-world, accented audio to bridge academic and industrial research.

Figures (1)

• Figure 1: WER by Region and Provider