AraToken: Optimizing Arabic Tokenization with Normalization Pipeline and Language Extension for Qwen3

TL;DR

AraToken addresses the tokenization bottleneck for Arabic by introducing a normalization-forwarded SentencePiece Unigram tokenizer and a Language Extension Pipeline (LEP) to fuse the tokenizer with Qwen3-0.6B. The normalization pipeline reduces orthographic variability (Alif variants, diacritics, numerals, Tatweel), yielding 18% lower fertility and superior compression. LEP enables rapid adaptation through mean-subtoken embedding initialization, gradient masking, and selective unfreezing, cutting evaluation loss from 8.28 to 2.43 within 800 steps on 100K Arabic samples. The work provides tokenizer, scripts, and checkpoints to spur Arabic NLP research and demonstrates practical pathways to extend multilingual models with language-specific tokenizers. Overall, AraToken achieves efficient Arabic tokenization and effective model adaptation with modest training budgets, offering a blueprint for similar language-extend pipelines.

Abstract

Tokenization is a critical preprocessing step for large language models (LLMs), directly impacting training efficiency and downstream performance. General-purpose tokenizers trained predominantly on English and Latin-script languages exhibit suboptimal performance on morphologically rich languages such as Arabic, resulting in inflated token sequences and reduced compression efficiency. In this work, we present AraToken, an Arabic-optimized tokenizer built on SentencePiece Unigram algorithm with a comprehensive normalization pipeline addressing Arabic-specific orthographic variations including Alif variants, diacritics, and Arabic-Indic numerals. We systematically compare BPE, WordPiece, and SentencePiece algorithms across multiple configurations, demonstrating that SentencePiece with normalization achieves 18% lower fertility (1.199 vs 1.35 tokens/word) compared to unnormalized baselines. Furthermore, we introduce the Language Extension Pipeline (LEP), a method for integrating the optimized tokenizer into Qwen3-0.6B through vocabulary extension with mean subtoken initialization and selective transformer layer unfreezing. Our experiments show that LEP reduces evaluation loss from 8.28 to 2.43 within 800 training steps on 100K Arabic samples. We release our tokenizer, training scripts, and model checkpoints to facilitate Arabic NLP research.

Figures (8)

• Figure 1: Overview of the AraToken pipeline. Arabic text is normalized and used to train a SentencePiece tokenizer, which is then integrated into Qwen3 via the Language Extension Pipeline (LEP).
• Figure 2: Examples of Arabic text normalization. The pipeline unifies Alif variants, converts Arabic-Indic numerals to Western digits, and removes Tatweel (text justification) characters.
• Figure 3: Language Extension Pipeline (LEP) architecture. Arabic tokens from the SentencePiece model are added to the Qwen3 vocabulary. New embeddings are initialized using mean subtoken initialization. During training, original embeddings are frozen via gradient masking, while the last 4 transformer layers are unfrozen for adaptation.
• Figure 4: Fertility comparison across tokenization algorithms. Lower fertility indicates more efficient encoding. Normalization consistently reduces fertility by 8-9% across all algorithms.
• Figure 5: Compression ratio comparison across tokenization algorithms. Higher compression indicates fewer tokens per character. SentencePiece with normalization achieves the best compression (5.03 chars/token).