101 Billion Arabic Words Dataset

TL;DR

The paper presents the 101 Billion Arabic Words Dataset, a large-scale Arabic corpus built from Common Crawl WET content to address data scarcity and biases in Arabic LLMs. It details a full pipeline—from data collection and raw extraction to extensive cleaning, deduplication, and tooling—grounded in a RefinedWeb-inspired approach but tailored to Arabic morphology and content. Key contributions include a six-split collection from Common Crawl, a multi-layer cleaning and deduplication process using Yamane sampling, URL filtering, MinHash-based deduplication, and standardized preprocessing with Camel tools and Tnkeeh, culminating in a 0.4 TB, 89.1M-page dataset of 101B words, now accessible on HuggingFace. The work highlights practical considerations for building authentic Arabic NLP resources, discusses limitations such as the lack of end-to-end model evaluation, and outlines future directions for assessing diversity and linguistic coverage to support more culturally attuned Arabic LLMs.

Abstract

In recent years, Large Language Models have revolutionized the field of natural language processing, showcasing an impressive rise predominantly in English-centric domains. These advancements have set a global benchmark, inspiring significant efforts toward developing Arabic LLMs capable of understanding and generating the Arabic language with remarkable accuracy. Despite these advancements, a critical challenge persists: the potential bias in Arabic LLMs, primarily attributed to their reliance on datasets comprising English data that has been translated into Arabic. This reliance not only compromises the authenticity of the generated content but also reflects a broader issue -the scarcity of original quality Arabic linguistic data. This study aims to address the data scarcity in the Arab world and to encourage the development of Arabic Language Models that are true to both the linguistic and nuances of the region. We undertook a large-scale data mining project, extracting a substantial volume of text from the Common Crawl WET files, specifically targeting Arabic content. The extracted data underwent a rigorous cleaning and deduplication process, using innovative techniques to ensure the integrity and uniqueness of the dataset. The result is the 101 Billion Arabic Words Dataset, the largest Arabic dataset available to date, which can significantly contribute to the development of authentic Arabic LLMs. This study not only highlights the potential for creating linguistically and culturally accurate Arabic LLMs but also sets a precedent for future research in enhancing the authenticity of Arabic language models.

Figures (2)

• Figure 1: JSONL file fields.
• Figure 2: Dataset cleaning pipeline