Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AraModernBERT: Transtokenized Initialization and Long-Context Encoder Modeling for Arabic

Omar Elshehy, Omer Nacar, Abdelbasset Djamai, Muhammed Ragab, Khloud Al Jallad, Mona Abdelazim

TL;DR

It is shown that transtokenization is essential for Arabic language modeling, yielding dramatic improvements in masked language modeling performance compared to non-transtokenized initialization, and that AraModernBERT supports stable and effective long-context modeling, achieving improved intrinsic language modeling performance at extended sequence lengths.

Abstract

Encoder-only transformer models remain widely used for discriminative NLP tasks, yet recent architectural advances have largely focused on English. In this work, we present AraModernBERT, an adaptation of the ModernBERT encoder architecture to Arabic, and study the impact of transtokenized embedding initialization and native long-context modeling up to 8,192 tokens. We show that transtokenization is essential for Arabic language modeling, yielding dramatic improvements in masked language modeling performance compared to non-transtokenized initialization. We further demonstrate that AraModernBERT supports stable and effective long-context modeling, achieving improved intrinsic language modeling performance at extended sequence lengths. Downstream evaluations on Arabic natural language understanding tasks, including inference, offensive language detection, question-question similarity, and named entity recognition, confirm strong transfer to discriminative and sequence labeling settings. Our results highlight practical considerations for adapting modern encoder architectures to Arabic and other languages written in Arabic-derived scripts.

AraModernBERT: Transtokenized Initialization and Long-Context Encoder Modeling for Arabic

TL;DR

It is shown that transtokenization is essential for Arabic language modeling, yielding dramatic improvements in masked language modeling performance compared to non-transtokenized initialization, and that AraModernBERT supports stable and effective long-context modeling, achieving improved intrinsic language modeling performance at extended sequence lengths.

Abstract

Encoder-only transformer models remain widely used for discriminative NLP tasks, yet recent architectural advances have largely focused on English. In this work, we present AraModernBERT, an adaptation of the ModernBERT encoder architecture to Arabic, and study the impact of transtokenized embedding initialization and native long-context modeling up to 8,192 tokens. We show that transtokenization is essential for Arabic language modeling, yielding dramatic improvements in masked language modeling performance compared to non-transtokenized initialization. We further demonstrate that AraModernBERT supports stable and effective long-context modeling, achieving improved intrinsic language modeling performance at extended sequence lengths. Downstream evaluations on Arabic natural language understanding tasks, including inference, offensive language detection, question-question similarity, and named entity recognition, confirm strong transfer to discriminative and sequence labeling settings. Our results highlight practical considerations for adapting modern encoder architectures to Arabic and other languages written in Arabic-derived scripts.
Paper Structure (20 sections, 1 equation, 1 figure, 6 tables)

This paper contains 20 sections, 1 equation, 1 figure, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Context Modeling.
  5. Arabic Tokenization.
  6. Transtokenized Embedding Initialization.
  7. Training Objective and Data.
  8. Experiments and Results
  9. Experimental Setup
  10. Evaluation Metrics
  11. Intrinsic Evaluation: Transtokenization Ablation
  12. Long-Context Language Modeling
  13. Arabic Natural Language Understanding
  14. Arabic Retrieval
  15. Short-Text Retrieval.
  16. ...and 5 more sections

Figures (1)

  • Figure 1: AraModernBERT integrates an Arabic BPE tokenizer with transtokenized embedding initialization and a ModernBERT encoder supporting native long-context modeling up to 8,192 tokens.