Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GemMaroc: Unlocking Darija Proficiency in LLMs with Minimal Data

Abderrahman Skiredj, Ferdaous Azhari, Houdaifa Atou, Nouamane Tazi, Ismail Berrada

TL;DR

GemMaroc demonstrates that Moroccan Arabic (Darija) proficiency can be unlocked in open LLMs with minimal, high-quality data and a green compute footprint. By translating three instruction suites (LIMA-1K, DEITA-6K, and a 44K reasoning slice of TULU-50K) into Darija and applying LoRA fine-tuning on Gemma bases, the approach achieves strong Darija benchmarks while preserving cross-lingual reasoning; the 27B model reaches 61.6% on DarijaMMLU and 60.5% on DarijaHellaSwag, at roughly 2% of the energy cost of prior work. The work provides open-source models, data, and code, enabling broader deployment in education, public services, and digital interaction in the Maghreb, and sets a blueprint for low-resource dialects to be both high-performing and sustainable.

Abstract

Open-source large language models (LLMs) still marginalise Moroccan Arabic (Darija), forcing practitioners either to bolt on heavyweight Arabic adapters or to sacrifice the very reasoning skills that make LLMs useful. We show that a rigorously quality-over-quantity alignment strategy can surface fluent Darija while safeguarding the backbone s cross-lingual reasoning at a sliver of the usual compute. We translate three compact instruction suites LIMA 1 K, DEITA 6 K and TULU 50 K into Darija, preserve 20 of the English originals, and add mathematics, coding and scientific prompts. A LoRA-tuned Gemma 3-4B trained on 5 K mixed instructions lifts DarijaMMLU from 32.8 to 42.7 ; adding the reasoning-dense TULU portion pushes it to 47.5 with no English regression. Scaling the identical recipe to Gemma 3-27B produces GemMaroc-27B, which matches Atlas-Chat on DarijaMMLU (61.6 ) and leaps ahead on Darija commonsense, scoring 60.5 on HellaSwag versus Atlas-Chat s 48.4 . Crucially, GemMaroc retains Gemma-27B s strong maths and general-reasoning ability, showing only minimal movement on GSM8K and English benchmarks. The entire model is trained in just 48 GPU.h, underscoring a Green AI pathway to inclusive, sustainable language technology. We release code, data and checkpoints to spur Darija-centric applications in education, public services and everyday digital interaction.

GemMaroc: Unlocking Darija Proficiency in LLMs with Minimal Data

TL;DR

GemMaroc demonstrates that Moroccan Arabic (Darija) proficiency can be unlocked in open LLMs with minimal, high-quality data and a green compute footprint. By translating three instruction suites (LIMA-1K, DEITA-6K, and a 44K reasoning slice of TULU-50K) into Darija and applying LoRA fine-tuning on Gemma bases, the approach achieves strong Darija benchmarks while preserving cross-lingual reasoning; the 27B model reaches 61.6% on DarijaMMLU and 60.5% on DarijaHellaSwag, at roughly 2% of the energy cost of prior work. The work provides open-source models, data, and code, enabling broader deployment in education, public services, and digital interaction in the Maghreb, and sets a blueprint for low-resource dialects to be both high-performing and sustainable.

Abstract

Open-source large language models (LLMs) still marginalise Moroccan Arabic (Darija), forcing practitioners either to bolt on heavyweight Arabic adapters or to sacrifice the very reasoning skills that make LLMs useful. We show that a rigorously quality-over-quantity alignment strategy can surface fluent Darija while safeguarding the backbone s cross-lingual reasoning at a sliver of the usual compute. We translate three compact instruction suites LIMA 1 K, DEITA 6 K and TULU 50 K into Darija, preserve 20 of the English originals, and add mathematics, coding and scientific prompts. A LoRA-tuned Gemma 3-4B trained on 5 K mixed instructions lifts DarijaMMLU from 32.8 to 42.7 ; adding the reasoning-dense TULU portion pushes it to 47.5 with no English regression. Scaling the identical recipe to Gemma 3-27B produces GemMaroc-27B, which matches Atlas-Chat on DarijaMMLU (61.6 ) and leaps ahead on Darija commonsense, scoring 60.5 on HellaSwag versus Atlas-Chat s 48.4 . Crucially, GemMaroc retains Gemma-27B s strong maths and general-reasoning ability, showing only minimal movement on GSM8K and English benchmarks. The entire model is trained in just 48 GPU.h, underscoring a Green AI pathway to inclusive, sustainable language technology. We release code, data and checkpoints to spur Darija-centric applications in education, public services and everyday digital interaction.

Paper Structure

This paper contains 34 sections, 6 equations, 7 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Contributions
  3. Related Work
  4. Darija and Arabic LLMs
  5. Improving Multilingual and Dialectal Capabilities in LLMs
  6. Minimal Data for SFT in Low-Resource Multilingual Settings
  7. The Importance of Multilingual Chain-of-Thought and Reasoning Samples
  8. GemMaroc: Our Fine-Tuning Approach to Unlock Darija Proficiency in Gemma LLMs with Minimal Resources
  9. SFT Data Preparation
  10. Model and Fine-Tuning Details
  11. Evaluation Benchmarks
  12. Experiments, Results, and Discussion
  13. Experimental Set-up
  14. Hardware and cost.
  15. Metrics.
  16. ...and 19 more sections

Figures (7)

  • Figure 1: Illustrative example 1 from Darija Lima dataset.
  • Figure 2: Illustrative example 2 from Darija Lima dataset.
  • Figure 3: Round 1 of an Illustrative example from Darija Deita dataset.
  • Figure 4: Round 2 of an Illustrative example from Darija Deita dataset.
  • Figure 5: Round 3 of an Illustrative example from Darija Deita dataset.
  • ...and 2 more figures