Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Doing More with Less: Data Augmentation for Sudanese Dialect Automatic Speech Recognition

Ayman Mansour

TL;DR

This work tackles Sudanese dialect ASR under severe data scarcity by systematically evaluating Whisper model variants and combining data augmentation strategies. It demonstrates that a hybrid approach—self-training with pseudo-labels plus TTS-based augmentation, anchored by a clean gold standard—yields the best generalization, achieving a holdout $WER$ of $51.6\%$ and $CER$ of $26.5\%$ with 28.37 hours of training. The study provides the first Sudanese dialect benchmark, reveals important generalization gaps between in-domain and holdout data, and offers a practical, resource-conscious roadmap for developing low-resource Arabic dialect ASR. These findings have practical impact for deploying robust dialectal ASR in settings with limited labeled data and emphasize the value of combining multiple data sources and augmentation techniques while highlighting ethical and dataset limitations.

Abstract

Although many Automatic Speech Recognition (ASR) systems have been developed for Modern Standard Arabic (MSA) and Dialectal Arabic (DA), few studies have focused on dialect-specific implementations, particularly for low-resource Arabic dialects such as Sudanese. This paper presents a comprehensive study of data augmentation techniques for fine-tuning OpenAI Whisper models and establishes the first benchmark for the Sudanese dialect. Two augmentation strategies are investigated: (1) self-training with pseudo-labels generated from unlabeled speech, and (2) TTS-based augmentation using synthetic speech from the Klaam TTS system. The best-performing model, Whisper-Medium fine-tuned with combined self-training and TTS augmentation (28.4 hours), achieves a Word Error Rate (WER) of 57.1% on the evaluation set and 51.6% on an out-of-domain holdout set substantially outperforming zero-shot multilingual Whisper (78.8% WER) and MSA-specialized Arabic models (73.8-123% WER). All experiments used low-cost resources (Kaggle free tier and Lightning.ai trial), demonstrating that strategic data augmentation can overcome resource limitations for low-resource dialects and provide a practical roadmap for developing ASR systems for low-resource Arabic dialects and other marginalized language varieties. The models, evaluation benchmarks, and reproducible training pipelines are publicly released to facilitate future research on low-resource Arabic ASR.

Doing More with Less: Data Augmentation for Sudanese Dialect Automatic Speech Recognition

TL;DR

This work tackles Sudanese dialect ASR under severe data scarcity by systematically evaluating Whisper model variants and combining data augmentation strategies. It demonstrates that a hybrid approach—self-training with pseudo-labels plus TTS-based augmentation, anchored by a clean gold standard—yields the best generalization, achieving a holdout of and of with 28.37 hours of training. The study provides the first Sudanese dialect benchmark, reveals important generalization gaps between in-domain and holdout data, and offers a practical, resource-conscious roadmap for developing low-resource Arabic dialect ASR. These findings have practical impact for deploying robust dialectal ASR in settings with limited labeled data and emphasize the value of combining multiple data sources and augmentation techniques while highlighting ethical and dataset limitations.

Abstract

Although many Automatic Speech Recognition (ASR) systems have been developed for Modern Standard Arabic (MSA) and Dialectal Arabic (DA), few studies have focused on dialect-specific implementations, particularly for low-resource Arabic dialects such as Sudanese. This paper presents a comprehensive study of data augmentation techniques for fine-tuning OpenAI Whisper models and establishes the first benchmark for the Sudanese dialect. Two augmentation strategies are investigated: (1) self-training with pseudo-labels generated from unlabeled speech, and (2) TTS-based augmentation using synthetic speech from the Klaam TTS system. The best-performing model, Whisper-Medium fine-tuned with combined self-training and TTS augmentation (28.4 hours), achieves a Word Error Rate (WER) of 57.1% on the evaluation set and 51.6% on an out-of-domain holdout set substantially outperforming zero-shot multilingual Whisper (78.8% WER) and MSA-specialized Arabic models (73.8-123% WER). All experiments used low-cost resources (Kaggle free tier and Lightning.ai trial), demonstrating that strategic data augmentation can overcome resource limitations for low-resource dialects and provide a practical roadmap for developing ASR systems for low-resource Arabic dialects and other marginalized language varieties. The models, evaluation benchmarks, and reproducible training pipelines are publicly released to facilitate future research on low-resource Arabic ASR.
Paper Structure (24 sections, 4 figures, 2 tables)

This paper contains 24 sections, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Datasets
  4. Sudanese Dialect Speech Dataset
  5. Arabic Speech Corpus
  6. Lisan-Sudanese TTS Dataset
  7. OOOK-Eval Dataset
  8. Method
  9. Whisper Models evaluation
  10. Whisper Models Fine-tuning
  11. Self-Training Implementation
  12. TTS Data Augmentation
  13. Evaluation
  14. Results and discussion
  15. Zero-shot Performance
  16. ...and 9 more sections

Figures (4)

  • Figure 1: Lisan-Sudanese TTS Dataset construction pipeline, detailing the transition from morphological tokens to synthetic audio generation.
  • Figure 2: Language Detection Failures
  • Figure 3: Error Type Distribution
  • Figure 4: Character Confusions