Improving the Inclusivity of Dutch Speech Recognition by Fine-tuning Whisper on the JASMIN-CGN Corpus

TL;DR

This study addresses inclusivity gaps in Dutch ASR by fine-tuning Whisper on the JASMIN-CGN corpus to handle age and nativeness variety. It leverages a rigorously designed 10-fold cross-validation scheme and subpopulation-specific as well as full-dataset fine-tuning to evaluate performance via Word Error Rate (WER). The results show substantial improvements over zero-shot performance, with the largest gains for native children, and demonstrate cross-group transfer effects that favor broader, multi-group training. The findings underscore the value of spontaneous, interaction-based Dutch speech data for robust, inclusive ASR and provide practical guidance for deploying models that perform well across diverse user populations.

Abstract

We test and study the variation in speech recognition of fine-tuned versions of the Whisper model on child, elderly and non-native Dutch speech from the JASMIN-CGN corpus. Our primary goal is to evaluate how speakers' age and linguistic background influence Whisper's performance. Whisper achieves varying Word Error Rates (WER) when fine-tuned on subpopulations of specific ages and linguistic backgrounds. Fine-tuned performance is remarkably better than zero-shot performance, achieving a relative reduction in WER of 81% for native children, 72% for non-native children, 67% for non-native adults, and 65% for native elderly people. Our findings underscore the importance of training speech recognition models like Whisper on underrepresented subpopulations such as children, the elderly, and non-native speakers.