TabiBERT: A Large-Scale ModernBERT Foundation Model and Unified Benchmarking Framework for Turkish

TL;DR

TabiBERT addresses two gaps in Turkish NLP: lack of a monolingual Turkish encoder that incorporates ModernBERT-style efficiency and long-context modeling, and the absence of a standardized Turkish benchmarking framework. Built on ModernBERT, TabiBERT is pre-trained from scratch on a large, diverse Turkish corpus totaling $8.658\times 10^{10}$ tokens (with $13\%$ non-Turkish data) and supports an $8192$-token context window, achieving up to $2.65\times$ faster inference and reduced memory footprint. The authors introduce TabiBench, a unified evaluation suite with 28 datasets across 8 task categories, enabling reproducible, cross-model comparisons and revealing state-of-the-art performance among Turkish encoders on five of eight categories (notably in QA and retrieval). They provide open-source model weights, training configs, and evaluation code, enabling broader adoption and further research, while discussing limitations and future directions such as scaling to larger models and expanding long-context Turkish tasks. Overall, TabiBERT demonstrates strong cross-domain generalization within Turkish NLP, offering a practical, efficient encoder with a transparent benchmarking framework that advances Turkish language understanding and retrieval-oriented applications.

Abstract

Since the inception of BERT, encoder-only Transformers have evolved significantly in computational efficiency, training stability, and long-context modeling. ModernBERT consolidates these advances by integrating Rotary Positional Embeddings (RoPE), FlashAttention, and refined normalization. Despite these developments, Turkish NLP lacks a monolingual encoder trained from scratch incorporating such modern architectural paradigms. This work introduces TabiBERT, a monolingual Turkish encoder based on ModernBERT architecture trained from scratch on a large, curated corpus. TabiBERT is pre-trained on one trillion tokens sampled from an 84.88B token multi-domain corpus: web text (73%), scientific publications (20%), source code (6%), and mathematical content (0.3%). The model supports 8,192-token context length (16x original BERT), achieves up to 2.65x inference speedup, and reduces GPU memory consumption, enabling larger batch sizes. We introduce TabiBench with 28 datasets across eight task categories with standardized splits and protocols, evaluated using GLUE-style macro-averaging. TabiBERT attains 77.58 on TabiBench, outperforming BERTurk by 1.62 points and establishing state-of-the-art on five of eight categories: question answering (+9.55), code retrieval (+2.41), and document retrieval (+0.60). Compared with task-specific prior best results, including specialized models like TurkishBERTweet, TabiBERT achieves +1.47 average improvement, indicating robust cross-domain generalization. We release model weights, training configurations, and evaluation code for transparent, reproducible Turkish encoder research.

Figures (1)

• Figure 1: Tokenizer fertility values of established Turkish BERT models and the modern multilingual mmBERT, relative to the proposed TabiBERT (represented by the dashed baseline). X axis labels indicate models along with the vocabulary sizes in parentheses. Y axis indicates relative number of tokens (fertility) required to represent Turkish texts compared to TabiBERT. Despite its large vocabulary of 256000.0 tokens, mmBERT suffers a 41.0% increase in fertility compared to TabiBERT in Turkish texts This demonstrates that despite architectural improvements, the lack of language-specific tokenizer optimization in mmBERT results in substantial subword fragmentation compared to TabiBERT.