Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Continued Pretraining for Low-Resource Swahili ASR: Achieving State-of-the-Art Performance with Minimal Labeled Data

Hillary Mutisya, John Mugane

Abstract

We investigate continued pretraining (CPT) for adapting wav2vec2-bert-2.0 to Swahili automatic speech recognition (ASR). Our approach combines unlabeled audio with limited labeled data through pseudo-labeled CPT followed by supervised finetuning. With 20,000 labeled samples, we achieve 3.24% WER on Common Voice Swahili-an 82% relative improvement over the baseline. This result surpasses the best previously reported academic system (8.3% WER from XLS-R) by 61% relative improvement. We provide concrete data requirements and a replicable methodology applicable to other low-resource languages.

Continued Pretraining for Low-Resource Swahili ASR: Achieving State-of-the-Art Performance with Minimal Labeled Data

Abstract

We investigate continued pretraining (CPT) for adapting wav2vec2-bert-2.0 to Swahili automatic speech recognition (ASR). Our approach combines unlabeled audio with limited labeled data through pseudo-labeled CPT followed by supervised finetuning. With 20,000 labeled samples, we achieve 3.24% WER on Common Voice Swahili-an 82% relative improvement over the baseline. This result surpasses the best previously reported academic system (8.3% WER from XLS-R) by 61% relative improvement. We provide concrete data requirements and a replicable methodology applicable to other low-resource languages.
Paper Structure (35 sections, 1 figure, 2 tables)

This paper contains 35 sections, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. The Data Challenge
  3. The CPT Hypothesis
  4. Research Questions
  5. Key Contributions
  6. Organization
  7. Related Work
  8. Self-Supervised Models
  9. Continued Pretraining
  10. Swahili ASR Benchmarks
  11. Positioning of This Work
  12. Methodology
  13. Experimental Design
  14. Model Architecture
  15. Datasets
  16. ...and 20 more sections

Figures (1)

  • Figure 1: Wav2vec2-BERT 2.0 architecture showing the flow from raw audio input through convolutional feature extraction, transformer-based contextual encoding, and task-specific CTC head for ASR. The model was pretrained on 4.5M hours across 104 languages including Swahili.