Pretraining Finnish ModernBERTs

TL;DR

The paper presents six Finnish ModernBERT encoder models (51M–475M) trained with a regionally focused multilingual corpus to handle Finnish morphology and long contexts beyond 512 tokens. It details a full pretraining pipeline on the LUMI supercomputer, including RoPE-based architectural choices that enable longer context, a three-phase training regimen, and carefully designed data-annealing strategies. Empirical evaluations across short-context EuroEval and long-context/CodeBEIR tasks show these models are competitive with, and often superior to, existing multilingual models, especially for long-context retrieval, while highlighting nuanced effects of annealing data and certain limitations in token-level tasks. The work also emphasizes eco-conscious design and release of models and code to the community, supporting further research and practical deployment in Finland and similarly multilingual regions.

Abstract

This paper reports on pretraining ModernBERT encoder models in six different sizes, ranging from 51M to 475M parameters, with a focus on limited multilingualism, emphasizing languages relevant to Finland. Our models are competitive with, or superior to, existing multilingual models. They outperform monolingual models on tasks that require a context longer than 512 tokens. We present empirical results on using different data in the final stage of training. The code and models are publicly released.

Figures (1)

• Figure 1: Larger vocabularies predictably have better fertilities compared to smaller ones.