Whisper Turns Stronger: Augmenting Wav2Vec 2.0 for Superior ASR in Low-Resource Languages

TL;DR

The paper tackles ASR for low-resource languages by augmenting the fine-tuning of Wav2Vec2 on synthetic data generated via Band-Stop, Gaussian-Noise, and Pitch-Shift transformations. Evaluated on Mozilla Common Voice data for Arabic, Russian, and Portuguese, the approach achieves substantial reductions in both Word Error Rate ($WER$) and Character Error Rate ($CER$) compared to baselines including Whisper and vanilla Wav2Vec2, with average improvements of approximately 33.9% in $WER$ and 53.2% in $CER$. The method demonstrates robustness to dialectal variation and Arabic diacritics, yielding the best results when all three augmentations are combined. These findings suggest that carefully designed data augmentation can bridge the gap caused by limited labeled data in under-represented languages and provide a path toward more inclusive, accurate ASR systems in real-world multilingual settings.

Abstract

Approaching Speech-to-Text and Automatic Speech Recognition problems in low-resource languages is notoriously challenging due to the scarcity of validated datasets and the diversity of dialects. Arabic, Russian, and Portuguese exemplify these difficulties, being low-resource languages due to the many dialects of these languages across different continents worldwide. Moreover, the variety of accents and pronunciations of such languages complicate ASR models' success. With the increasing popularity of Deep Learning and Transformers, acoustic models like the renowned Wav2Vec2 have achieved superior performance in the Speech Recognition field compared to state-of-the-art approaches. However, despite Wav2Vec2's improved efficiency over traditional methods, its performance significantly declines for under-represented languages, even though it requires significantly less labeled data. This paper introduces an end-to-end framework that enhances ASR systems fine-tuned on Wav2Vec2 through data augmentation techniques. To validate our framework's effectiveness, we conducted a detailed experimental evaluation using three datasets from Mozilla's Common Voice project in Arabic, Russian, and Portuguese. Additionally, the framework presented in this paper demonstrates robustness to different diacritics. Ultimately, our approach outperforms two previous baseline models, which are the pre-trained Wav2Vec2 and the well-known Whisper ASR model, resulting in an average relative improvement of 33.9\% in Word Error Rate and a 53.2\% relative improvement in Character Error Rate.

Figures (3)

• Figure 1: The framework presented in this paper.
• Figure 2: An example of two identical sentences using different diacritics.
• Figure 3: CER bar-chart for different combinations of augmentation methods. It is clear that the more augmentations used, the better the result. The best combination is the use of all three augmentations: Pitch-Shift, Gaussian-Noise, and Band-Stop, achieving a 19% CER for a variety of diacritics in Arabic, which triples the size of the letters vocabulary to 80 letters.