Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EuroBERT: Scaling Multilingual Encoders for European Languages

Nicolas Boizard, Hippolyte Gisserot-Boukhlef, Duarte M. Alves, André Martins, Ayoub Hammal, Caio Corro, Céline Hudelot, Emmanuel Malherbe, Etienne Malaboeuf, Fanny Jourdan, Gabriel Hautreux, João Alves, Kevin El-Haddad, Manuel Faysse, Maxime Peyrard, Nuno M. Guerreiro, Patrick Fernandes, Ricardo Rei, Pierre Colombo

TL;DR

EuroBERT revisits general-purpose multilingual encoders by incorporating decoder-inspired architectural and training innovations into a bidirectional encoder. It introduces a family of models (210M, 610M, 2.1B) trained on a 5T-token multilingual corpus that includes code and mathematics, with a two-phase training pipeline (pre-training and annealing) and long-context support up to 8,192 tokens. Across multilingual retrieval, classification, regression, and code/math tasks, EuroBERT achieves state-of-the-art or near-state-of-the-art results for its size, with notable strengths in code and mathematics domains and robust long-context performance, while revealing trade-offs between retrieval and classification. The work also provides extensive ablations on dataset composition and annealing strategies and releases models, intermediate checkpoints, and training framework to foster future research in multilingual encoders.

Abstract

General-purpose multilingual vector representations, used in retrieval, regression and classification, are traditionally obtained from bidirectional encoder models. Despite their wide applicability, encoders have been recently overshadowed by advances in generative decoder-only models. However, many innovations driving this progress are not inherently tied to decoders. In this paper, we revisit the development of multilingual encoders through the lens of these advances, and introduce EuroBERT, a family of multilingual encoders covering European and widely spoken global languages. Our models outperform existing alternatives across a diverse range of tasks, spanning multilingual capabilities, mathematics, and coding, and natively supporting sequences of up to 8,192 tokens. We also examine the design decisions behind EuroBERT, offering insights into our dataset composition and training pipeline. We publicly release the EuroBERT models, including intermediate training checkpoints, together with our training framework.

EuroBERT: Scaling Multilingual Encoders for European Languages

TL;DR

EuroBERT revisits general-purpose multilingual encoders by incorporating decoder-inspired architectural and training innovations into a bidirectional encoder. It introduces a family of models (210M, 610M, 2.1B) trained on a 5T-token multilingual corpus that includes code and mathematics, with a two-phase training pipeline (pre-training and annealing) and long-context support up to 8,192 tokens. Across multilingual retrieval, classification, regression, and code/math tasks, EuroBERT achieves state-of-the-art or near-state-of-the-art results for its size, with notable strengths in code and mathematics domains and robust long-context performance, while revealing trade-offs between retrieval and classification. The work also provides extensive ablations on dataset composition and annealing strategies and releases models, intermediate checkpoints, and training framework to foster future research in multilingual encoders.

Abstract

General-purpose multilingual vector representations, used in retrieval, regression and classification, are traditionally obtained from bidirectional encoder models. Despite their wide applicability, encoders have been recently overshadowed by advances in generative decoder-only models. However, many innovations driving this progress are not inherently tied to decoders. In this paper, we revisit the development of multilingual encoders through the lens of these advances, and introduce EuroBERT, a family of multilingual encoders covering European and widely spoken global languages. Our models outperform existing alternatives across a diverse range of tasks, spanning multilingual capabilities, mathematics, and coding, and natively supporting sequences of up to 8,192 tokens. We also examine the design decisions behind EuroBERT, offering insights into our dataset composition and training pipeline. We publicly release the EuroBERT models, including intermediate training checkpoints, together with our training framework.

Paper Structure

This paper contains 45 sections, 8 figures, 13 tables.

Table of Contents

  1. Introduction
  2. EuroBERT: A Refreshed Multilingual Encoder
  3. Architecture
  4. Dataset
  5. Pre-training mixture.
  6. Annealing mixture.
  7. Training Recipe
  8. Masked language modeling.
  9. Hyperparameters.
  10. Infrastructure.
  11. Evaluation
  12. Evaluation Setup
  13. Datasets and tasks.
  14. Baselines.
  15. Fine-tuning.
  16. ...and 30 more sections

Figures (8)

  • Figure 1: Pareto plots for multilingual tasks (top), showing retrieval performance on MIRACL and sentence classification on XNLI, and for math and code tasks (bottom), featuring CodeSearchNet and MathShepherd. The shaded regions indicate the Pareto frontiers.
  • Figure 2: Difference in F1 Score between EuroBERT and XLM-RoBERTa by tokenizer fertility (left plot) and fertility distribution for both tokenizers (right plot) on the NER dataset.
  • Figure 3: Results by length of the positive documents for retrieval (MLDR) and input documents for summarization (SeaHorse).
  • Figure 4: Impact of changing data subset ratios during annealing. The first vertical axis in each subplot denotes the reference data mix from \ref{['tab:xp_annealing_data']}.
  • Figure 5: Impact of hyperparameter choices during annealing. The first vertical axis in each subplot denotes the reference data mix from \ref{['tab:xp_annealing_data']}.
  • ...and 3 more figures