The Evolution of Darija Open Dataset: Introducing Version 2

TL;DR

Darija Open Dataset (DODa) addresses the under-resourced Moroccan dialect by releasing a large, open, bilingual dataset. Version 2 expands to roughly 100,000 entries and integrates both Arabic and Latin scripts, with semantic and syntactic annotations, verb conjugations, idioms, and name lexicons. The dataset supports translation tasks and NLP tool development, enabling cross-script analysis and community-driven expansion, including collaboration with AtlasIA. The authors also outline licensing under CC BY-NC 4.0 and plan an evaluation dataset to benchmark translation models, signaling a path toward robust, open, dialect-specific NLP resources.

Abstract

Darija Open Dataset (DODa) represents an open-source project aimed at enhancing Natural Language Processing capabilities for the Moroccan dialect, Darija. With approximately 100,000 entries, DODa stands as the largest collaborative project of its kind for Darija-English translation. The dataset features semantic and syntactic categorizations, variations in spelling, verb conjugations across multiple tenses, as well as tens of thousands of translated sentences. The dataset includes entries written in both Latin and Arabic alphabets, reflecting the linguistic variations and preferences found in different sources and applications. The availability of such dataset is critical for developing applications that can accurately understand and generate Darija, thus supporting the linguistic needs of the Moroccan community and potentially extending to similar dialects in neighboring regions. This paper explores the strategic importance of DODa, its current achievements, and the envisioned future enhancements that will continue to promote its use and expansion in the global NLP landscape.

Figures (1)

• Figure 1: Distribution of Content Types in the DODa