Error-preserving Automatic Speech Recognition of Young English Learners' Language

TL;DR

The paper tackles preserving language-learner errors in automatic speech recognition (ASR) to enable effective corrective feedback in language-learning tools. It collects a novel 85-hour corpus of spontaneous Swiss English speech from 4th–6th graders, annotates errors, and defines the Word-Based Error Preservation Rate ($WEPR$) to quantify how well ASR transcripts retain learner errors. Through extensive experiments, it shows that fine-tuning a learner-data model (ChaLL-300M) yields superior error preservation compared with state-of-the-art pre-trained systems, while large encoder–decoder models like Whisper excel on standard metrics. The work demonstrates that learner-focused data and targeted fine-tuning are essential for practical, error-aware ASR in education, and outlines directions for improving robustness, including larger models and joint error-prediction with feedback delivery.

Abstract

One of the central skills that language learners need to practice is speaking the language. Currently, students in school do not get enough speaking opportunities and lack conversational practice. Recent advances in speech technology and natural language processing allow for the creation of novel tools to practice their speaking skills. In this work, we tackle the first component of such a pipeline, namely, the automated speech recognition module (ASR), which faces a number of challenges: first, state-of-the-art ASR models are often trained on adult read-aloud data by native speakers and do not transfer well to young language learners' speech. Second, most ASR systems contain a powerful language model, which smooths out errors made by the speakers. To give corrective feedback, which is a crucial part of language learning, the ASR systems in our setting need to preserve the errors made by the language learners. In this work, we build an ASR system that satisfies these requirements: it works on spontaneous speech by young language learners and preserves their errors. For this, we collected a corpus containing around 85 hours of English audio spoken by learners in Switzerland from grades 4 to 6 on different language learning tasks, which we used to train an ASR model. Our experiments show that our model benefits from direct fine-tuning on children's voices and has a much higher error preservation rate than other models.

Figures (3)

• Figure 1: Number of utterances (outer ring) and audio hours (inner ring) by school grade (a) and school area code (b).
• Figure 2: Distribution of utterance lengths.
• Figure 3: Duration and grade distribution of the data folds.