Alexandria: A Multi-Domain Dialectal Arabic Machine Translation Dataset for Culturally Inclusive and Linguistically Diverse LLMs

TL;DR

Alexandria tackles the MT gap caused by Arabic diglossia by delivering the first large-scale, city-aware dialectal Arabic dataset across 13 Arab countries and 11 domains, totaling 107K turns with gender-annotated, multi-turn dialogues. Constructed via a three-phase pipeline—English source generation with Gemini-2.5 Pro, meticulous human dialectal translation, and rigorous peer revision—the dataset supports both training and rigorous evaluation of dialect-aware MT and LLMs. Evaluation across 24 Arabic-capable LLMs reveals persistent gaps in dialect authenticity and semantic adequacy, with strong gender-adherence but varying coverage of sub-dialects and terminologies. Alexandria thus provides a robust benchmark and resource for developing culturally inclusive, dialect-sensitive language technologies in the Arab world, while highlighting areas for methodological improvement and broader representation in future work.

Abstract

Arabic is a highly diglossic language where most daily communication occurs in regional dialects rather than Modern Standard Arabic. Despite this, machine translation (MT) systems often generalize poorly to dialectal input, limiting their utility for millions of speakers. We introduce \textbf{Alexandria}, a large-scale, community-driven, human-translated dataset designed to bridge this gap. Alexandria covers 13 Arab countries and 11 high-impact domains, including health, education, and agriculture. Unlike previous resources, Alexandria provides unprecedented granularity by associating contributions with city-of-origin metadata, capturing authentic local varieties beyond coarse regional labels. The dataset consists of multi-turn conversational scenarios annotated with speaker-addressee gender configurations, enabling the study of gender-conditioned variation in dialectal use. Comprising 107K total samples, Alexandria serves as both a training resource and a rigorous benchmark for evaluating MT and Large Language Models (LLMs). Our automatic and human evaluation of Arabic-aware LLMs benchmarks current capabilities in translating across diverse Arabic dialects and sub-dialects, while exposing significant persistent challenges.

Figures (18)

• Figure 1: Geographic distribution of Alexandria project participants by city across the Arab world. Point diameter is proportional to participant volume. Representative examples (abbreviated to two-turn interactions) are provided to demonstrate the dataset’s coverage across diverse Arabic dialects, domains, and genders.
• Figure 2: The data creation workflow for the Alexandria dataset. The process illustrates three key phases: (i) English source generation, (ii) human translation into Dialectal Arabic, and (iii) peer-revision and correction.
• Figure 3: Context-aware MT performance (spBLEU) across 13 dialects. Results reveal a significant directional asymmetry: models perform consistently stronger on Dialect $\to$ English (right) than English $\to$ Dialect (left). Maghrebi dialects (e.g., MR, MA, TN) remain the most challenging across all models.
• Figure 4: Intra-country performance variance (English $\to$ Sub-Dialect). Scores for selected sub-dialects reveal systematic difficulty gaps within countries (e.g., urban vs. rural Palestinian varieties), with consistent model rankings across sub-dialects.
• Figure 5: Domain robustness analysis (English $\to$ Dialect). The plot illustrates spBLEU scores for a subset of models across all 11 domains, demonstrating consistent performance stratification regardless of the domain.