Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Ara-Best-RQ: Multi Dialectal Arabic SSL

Haroun Elleuch, Ryan Whetten, Salima Mdhaffar, Yannick Estève, Fethi Bougares

Abstract

We present Ara-BEST-RQ, a family of self-supervised learning (SSL) models specifically designed for multi-dialectal Arabic speech processing. Leveraging 5,640 hours of crawled Creative Commons speech and combining it with publicly available datasets, we pre-train conformer-based BEST-RQ models up to 600M parameters. Our models are evaluated on dialect identification (DID) and automatic speech recognition (ASR) tasks, achieving state-of-the-art performance on the former while using fewer parameters than competing models. We demonstrate that family-targeted pre-training on Arabic dialects significantly improves downstream performance compared to multilingual or monolingual models trained on non-Arabic data. All models, code, and pre-processed datasets will be publicly released to support reproducibility and further research in Arabic speech technologies.

Ara-Best-RQ: Multi Dialectal Arabic SSL

Abstract

We present Ara-BEST-RQ, a family of self-supervised learning (SSL) models specifically designed for multi-dialectal Arabic speech processing. Leveraging 5,640 hours of crawled Creative Commons speech and combining it with publicly available datasets, we pre-train conformer-based BEST-RQ models up to 600M parameters. Our models are evaluated on dialect identification (DID) and automatic speech recognition (ASR) tasks, achieving state-of-the-art performance on the former while using fewer parameters than competing models. We demonstrate that family-targeted pre-training on Arabic dialects significantly improves downstream performance compared to multilingual or monolingual models trained on non-Arabic data. All models, code, and pre-processed datasets will be publicly released to support reproducibility and further research in Arabic speech technologies.
Paper Structure (13 sections, 1 figure, 5 tables)

This paper contains 13 sections, 1 figure, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Dataset
  4. Crawled Dataset
  5. Combined Dataset
  6. Experiments & Results
  7. Ara-BEST-RQ pre-training
  8. Downstream Fine-tuning
  9. Automatic Speech Recognition
  10. Dialect Identification
  11. Limitations
  12. Conclusion
  13. Acknowledgements

Figures (1)

  • Figure 1: Distribution of the full training set (in hours) by dialect.