Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MrBERT: Modern Multilingual Encoders via Vocabulary, Domain, and Dimensional Adaptation

Daniel Tamayo, Iñaki Lacunza, Paula Rivera-Hidalgo, Severino Da Dalt, Javier Aula-Blasco, Aitor Gonzalez-Agirre, Marta Villegas

TL;DR

MrBERT, a family of 150M-300M parameter encoders built on the ModernBERT architecture and pre-trained on 35 languages and code, is introduced, demonstrating that modern encoder architectures can be optimized for both localized linguistic excellence and efficient, high-stakes domain specialization.

Abstract

We introduce MrBERT, a family of 150M-300M parameter encoders built on the ModernBERT architecture and pre-trained on 35 languages and code. Through targeted adaptation, this model family achieves state-of-the-art results on Catalan- and Spanish-specific tasks, while establishing robust performance across specialized biomedical and legal domains. To bridge the gap between research and production, we incorporate Matryoshka Representation Learning (MRL), enabling flexible vector sizing that significantly reduces inference and storage costs. Ultimately, the MrBERT family demonstrates that modern encoder architectures can be optimized for both localized linguistic excellence and efficient, high-stakes domain specialization. We open source the complete model family on Huggingface.

MrBERT: Modern Multilingual Encoders via Vocabulary, Domain, and Dimensional Adaptation

TL;DR

MrBERT, a family of 150M-300M parameter encoders built on the ModernBERT architecture and pre-trained on 35 languages and code, is introduced, demonstrating that modern encoder architectures can be optimized for both localized linguistic excellence and efficient, high-stakes domain specialization.

Abstract

We introduce MrBERT, a family of 150M-300M parameter encoders built on the ModernBERT architecture and pre-trained on 35 languages and code. Through targeted adaptation, this model family achieves state-of-the-art results on Catalan- and Spanish-specific tasks, while establishing robust performance across specialized biomedical and legal domains. To bridge the gap between research and production, we incorporate Matryoshka Representation Learning (MRL), enabling flexible vector sizing that significantly reduces inference and storage costs. Ultimately, the MrBERT family demonstrates that modern encoder architectures can be optimized for both localized linguistic excellence and efficient, high-stakes domain specialization. We open source the complete model family on Huggingface.
Paper Structure (40 sections, 4 figures, 13 tables)

This paper contains 40 sections, 4 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. The 'Extraction vs. Pre-training' Debate
  4. Adaptation and Multilinguality
  5. Flexible Representation Learning
  6. Pre-training and Adaptation
  7. Data
  8. Pre-Training
  9. Language Adaptation
  10. Domain Adaptation
  11. Classification of targeted domain instances
  12. Domain Mapping and Selection Logic
  13. Pre-Training Settings
  14. Language and Domain Specialization
  15. Vocabulary Adaptation and Initialization.
  16. ...and 25 more sections

Figures (4)

  • Figure 1: Token distribution per language for the Pre-Training phase. The table is shown in logarithmic format for visualization purposes.
  • Figure 2: MrBERT performance across XTREME, CLUB, and EvalES benchmarks comparing AttMAT (attention head pruning), MAT (MLP hidden size reduction), and standard models (100%). Only average scores shown.
  • Figure 3: Inference throughput of matryoshka variants (sequence length: 8,192 tokens).
  • Figure 4: Performance comparison of matryoshka models at different compression levels (25%, 50%, 75%, 100% of the attention heads) against MrBERT models without matryoshka training across four benchmark tasks. Bars represent the average performance difference to the previous higher benchmark value.