NeoDictaBERT: Pushing the Frontier of BERT models for Hebrew

TL;DR

This work addresses the performance gap between traditional BERT-style Hebrew models and modern encoder-decoder architectures by introducing NeoDictaBERT and NeoDictaBERT-bilingual, both built on the NeoBERT framework with an extended 4,096-token context. The authors detail architectural innovations (Pre-RMSNorm, RoPE, SwiGLU), a non-shared embedding/head, and a deeper 28-layer design, along with Hebrew-centric and bilingual pretraining data and tokenizers. Empirical results show substantial improvements over prior Hebrew models across parsing, NER, sentiment, and QA, with additional strong retrieval performance when the bilingual model is fine-tuned on English data. The models are released publicly to spur Hebrew NLP research and development, with demonstrated cross-lingual retrieval capabilities and practical relevance for downstream tasks.

Abstract

Since their initial release, BERT models have demonstrated exceptional performance on a variety of tasks, despite their relatively small size (BERT-base has ~100M parameters). Nevertheless, the architectural choices used in these models are outdated compared to newer transformer-based models such as Llama3 and Qwen3. In recent months, several architectures have been proposed to close this gap. ModernBERT and NeoBERT both show strong improvements on English benchmarks and significantly extend the supported context window. Following their successes, we introduce NeoDictaBERT and NeoDictaBERT-bilingual: BERT-style models trained using the same architecture as NeoBERT, with a dedicated focus on Hebrew texts. These models outperform existing ones on almost all Hebrew benchmarks and provide a strong foundation for downstream tasks. Notably, the NeoDictaBERT-bilingual model shows strong results on retrieval tasks, outperforming other multilingual models of similar size. In this paper, we describe the training process and report results across various benchmarks. We release the models to the community as part of our goal to advance research and development in Hebrew NLP.