ASR-FAIRBENCH: Measuring and Benchmarking Equity Across Speech Recognition Systems
Anand Rai, Satyam Rahangdale, Utkarsh Anand, Animesh Mukherjee
TL;DR
Addresses the problem of demographic disparities in ASR performance. The authors propose ASR-FAIRBENCH, a real-time fairness-aware leaderboard that couples accuracy (WER) with equity via a mixed-effects Poisson regression and the FAAS metric. Key contributions include a stratified 10% Fair-Speech sample to enable scalable evaluation, a five-tier fairness classification, and an interactive, reproducible web platform. The framework uses $FAAS = 10 \times \log_{10}\left(\frac{Overall\,Score}{WER}\right)$ to jointly optimize fairness and accuracy and reveals substantial demographic disparities among state-of-the-art systems.
Abstract
Automatic Speech Recognition (ASR) systems have become ubiquitous in everyday applications, yet significant disparities in performance across diverse demographic groups persist. In this work, we introduce the ASR-FAIRBENCH leaderboard which is designed to assess both the accuracy and equity of ASR models in real-time. Leveraging the Meta's Fair-Speech dataset, which captures diverse demographic characteristics, we employ a mixed-effects Poisson regression model to derive an overall fairness score. This score is integrated with traditional metrics like Word Error Rate (WER) to compute the Fairness Adjusted ASR Score (FAAS), providing a comprehensive evaluation framework. Our approach reveals significant performance disparities in SOTA ASR models across demographic groups and offers a benchmark to drive the development of more inclusive ASR technologies.