Automatic Speech Recognition for Non-Native English: Accuracy and Disfluency Handling
Michael McGuire
TL;DR
This study evaluates five modern ASR systems on non-native English across six L1 backgrounds using read and spontaneous speech from the L2-ARCTIC corpus. It analyzes accuracy via Match Error Rate (MER), processing time, and disfluency handling under retained and omitted conditions, revealing that Whisper and AssemblyAI excel on read speech, while RevAI leads on spontaneous narratives with disfluencies retained. The results show clear L1-related performance gaps and highlight system-specific strengths in disfluency categories, with no consistent speed-accuracy trade-off. The findings offer practical guidance for language instructors and researchers selecting ASR tools for read versus spontaneous learner tasks and underscore the value of open data and targeted fine-tuning to improve robustness for non-native speech.
Abstract
Automatic speech recognition (ASR) has been an essential component of computer assisted language learning (CALL) and computer assisted language testing (CALT) for many years. As this technology continues to develop rapidly, it is important to evaluate the accuracy of current ASR systems for language learning applications. This study assesses five cutting-edge ASR systems' recognition of non-native accented English speech using recordings from the L2-ARCTIC corpus, featuring speakers from six different L1 backgrounds (Arabic, Chinese, Hindi, Korean, Spanish, and Vietnamese), in the form of both read and spontaneous speech. The read speech consisted of 2,400 single sentence recordings from 24 speakers, while the spontaneous speech included narrative recordings from 22 speakers. Results showed that for read speech, Whisper and AssemblyAI achieved the best accuracy with mean Match Error Rates (MER) of 0.054 and 0.056 respectively, approaching human-level accuracy. For spontaneous speech, RevAI performed best with a mean MER of 0.063. The study also examined how each system handled disfluencies such as filler words, repetitions, and revisions, finding significant variation in performance across systems and disfluency types. While processing speed varied considerably between systems, longer processing times did not necessarily correlate with better accuracy. By detailing the performance of several of the most recent, widely-available ASR systems on non-native English speech, this study aims to help language instructors and researchers understand the strengths and weaknesses of each system and identify which may be suitable for specific use cases.