Whispering in Amharic: Fine-tuning Whisper for Low-resource Language

TL;DR

Addressing Amharic ASR, the paper fine-tunes Whisper-small-am on a mixture of Amharic datasets (FLEURS, Mozilla Common Voice, BDU-speech) to overcome Amharic underrepresentation. It demonstrates that combining existing and new data yields large improvements in WER, CER, and BLEU, while normalization of homophones further boosts evaluation metrics. Zero-shot experiments with pre-trained Whisper variants perform poorly on Amharic, underscoring the need for language-specific fine-tuning. The work highlights dataset composition and cross-domain robustness as critical for enabling practical ASR in low-resource languages and guides future improvements in Amharic speech technology.

Abstract

This work explores fine-tuning OpenAI's Whisper automatic speech recognition (ASR) model for Amharic, a low-resource language, to improve transcription accuracy. While the foundational Whisper model struggles with Amharic due to limited representation in its training data, we fine-tune it using datasets like Mozilla Common Voice, FLEURS, and the BDU-speech dataset. The best-performing model, Whispersmall-am, significantly improves when finetuned on a mix of existing FLEURS data and new, unseen Amharic datasets. Training solely on new data leads to poor performance, but combining it with FLEURS data reinforces the model, enabling better specialization in Amharic. We also demonstrate that normalizing Amharic homophones significantly enhances Word Error Rate (WER) and Bilingual Evaluation Understudy (BLEU) scores. This study underscores the importance of fine-tuning strategies and dataset composition for improving ASR in low-resource languages, providing insights for future Amharic speech recognition research.

Figures (1)

• Figure 1: The fine-tuning process