HalleluBERT: Let every token that has meaning bear its weight

TL;DR

This work addresses the lack of large-scale, monolingual Hebrew encoders by training HalleluBERTbase and HalleluBERTlarge, RoBERTa-style models from scratch on ~49.1 GB of Hebrew text with a Hebrew-specific 52k BPE. The authors demonstrate that fully converged monolingual pretraining yields state-of-the-art results on Hebrew NER (BMC, NEMO2) and sentiment (SMCD), outperforming both monolingual Hebrew and multilingual baselines. They provide a detailed training and evaluation protocol, including data curation (HeDC4 plus Wikipedia), a dedicated Hebrew tokenizer, and a robust evaluation framework aligned with the HeRo benchmarks. The results substantiate the value of large-batch, long-duration pretraining for Hebrew, with HalleluBERTlarge achieving the best overall average across tasks, and HalleluBERTbase showing strongest NER performance; this has practical implications for Hebrew NLP applications and downstream tasks requiring Hebrew understanding.

Abstract

Transformer-based models have advanced NLP, yet Hebrew still lacks a large-scale RoBERTa encoder which is extensively trained. Existing models such as HeBERT, AlephBERT, and HeRo are limited by corpus size, vocabulary, or training depth. We present HalleluBERT, a RoBERTa-based encoder family (base and large) trained from scratch on 49.1~GB of deduplicated Hebrew web text and Wikipedia with a Hebrew-specific byte-level BPE vocabulary. Evaluated on NER and sentiment classification benchmarks, HalleluBERT outperforms both monolingual and multilingual baselines. HalleluBERT sets a new state of the art for Hebrew and highlights the benefits of fully converged monolingual pretraining.

Figures (1)

• Figure 1: Perplexity of the HalleluBERT models. Top based on a validation at the checkpoints. Bottom based on the validation of each optimization cycle during the training.