Tahakom LLM Guidelines and Recipes: From Pre-training Data to an Arabic LLM

TL;DR

This work tackles the core challenges of building robust Arabic LLMs by focusing on high-quality data curation, tokenizer design, and culturally aware evaluation. It introduces a multi-stage data pipeline (CuAra) combining extraction, language filtering, heuristic quality filters, and model-based filtering to create a scalable Arabic pre-training corpus that outperforms existing open datasets. The authors systematically study tokenizer design, showing that vocabulary size, data-distribution alignment, and to a lesser extent pre-tokenization, meaningfully affect downstream Arabic tasks. They also advance Arabic evaluation by refining ARB-MMLU into Enhanced-ARB-MMLU and adding the Saudi Culture Dataset, accompanied by an Arabic LLM leaderboard and an answer-shuffling protocol to better gauge model reliability and cultural alignment. Together, these contributions advance reproducibility, efficiency, and culturally grounded benchmarking for Arabic LLMs, with CuAra and Enhanced-ARB-MMLU serving as practical resources for the community.

Abstract

Large Language Models (LLMs) have significantly advanced the field of natural language processing, enhancing capabilities in both language understanding and generation across diverse domains. However, developing LLMs for Arabic presents unique challenges. This paper explores these challenges by focusing on critical aspects such as data curation, tokenizer design, and evaluation. We detail our approach to the collection and filtration of Arabic pre-training datasets, assess the impact of various tokenizer designs on model performance, and examine the limitations of existing Arabic evaluation frameworks, for which we propose a systematic corrective methodology. To promote transparency and facilitate collaborative development, we share our data and methodologies, contributing to the advancement of language modeling, particularly for the Arabic language.

Figures (14)

• Figure 1: Overview of the data preparation pipeline, from data extraction to model-based filtering. WET files provide ready-to-use plain text, whereas WARC files require text extraction. After that, we apply the base filtration pipeline, followed by model-based filtering to obtain high-quality Arabic Pre-training data.
• Figure 2: The base filtration pipeline, composed of multiple filters in addition to deduplication (MinHash). Numbers indicate the percentage of web pages left after each filter.
• Figure 3: The Confusion matrix of the classes assigned by the educational classifier for both English and Arabic prompts on a 10K sample. We can see the agreement on most of the web pages, but generally, the Arabic prompt tends to give lower scores compared to the English prompt.
• Figure 4: The distribution of the classes assigned by the educational classifier for both English and Arabic prompts on a 10K sample. While both are right-skewed, the Arabic prompt assigned more 0 and 1 classes compared to the English prompt
• Figure 5: Comparison of fertility scores and vocabulary sizes (left) and downstream Arabic dataset accuracy and vocabulary sizes (right) for the pre-trained LLaMA3.2-1B model with a tokenizers trained on Arabic text. Fertility scores generally decrease with larger vocabularies; however, this does not consistently result in higher accuracy across all evaluated datasets