Clustering and Mining Accented Speech for Inclusive and Fair Speech Recognition
Jaeyoung Kim, Han Lu, Soheil Khorram, Anshuman Tripathi, Qian Zhang, Hasim Sak
TL;DR
The paper addresses biased ASR performance on accented speech due to uneven accent representation. It proposes a comprehensive accent recognition framework that combines a conv-Transformer architecture with three strategies: supervised/unsupervised pre-training, distributionally robust optimization, and online K-means clustering to handle limited and imbalanced data. By fine-tuning a Transformer-Transducer ASR on mined accented speech, it achieves notable relative improvements on underrepresented accents (e.g., Indian) and reduces performance variance across accents. The approach offers a path toward fairer, more inclusive speech recognition while reducing data collection demands for rare accents.
Abstract
Modern automatic speech recognition (ASR) systems are typically trained on more than tens of thousands hours of speech data, which is one of the main factors for their great success. However, the distribution of such data is typically biased towards common accents or typical speech patterns. As a result, those systems often poorly perform on atypical accented speech. In this paper, we present accent clustering and mining schemes for fair speech recognition systems which can perform equally well on under-represented accented speech. For accent recognition, we applied three schemes to overcome limited size of supervised accent data: supervised or unsupervised pre-training, distributionally robust optimization (DRO) and unsupervised clustering. Three schemes can significantly improve the accent recognition model especially for unbalanced and small accented speech. Fine-tuning ASR on the mined Indian accent speech using the proposed supervised or unsupervised clustering schemes showed 10.0% and 5.3% relative improvements compared to fine-tuning on the randomly sampled speech, respectively.