New Encoders for German Trained from Scratch: Comparing ModernGBERT with Converted LLM2Vec Models

TL;DR

This work systematically compares two practical paths to high-quality German encoders under identical data and training constraints: training from scratch (ModernGBERT) and converting decoders into encoders via LLM2Vec (LLäMmlein2Vec). It introduces context-extension to 8192 tokens and presents two resources, ModernGBERT and LLäMmlein2Vec, along with an encoder-adapted QA-NIAH benchmark and full training transparency. The results show that from-scratch ModernGBERT, particularly the 1B variant, achieves state-of-the-art performance on SuperGLEBer and strong results on MTEB after fine-tuning, while LLäMmlein2Vec provides competitive performance with much lower training cost, especially when a suitable pre-trained decoder exists. Practically, the paper provides actionable guidance: use from-scratch encoders when parameter efficiency and latency matter, and prefer conversion when pre-trained decoders are available and compute is constrained; all models and training artifacts are released under a research-only license to foster further work.

Abstract

Encoders remain essential for efficient German NLP and NLU scenarios despite the rise of decoder-only LLMs. This work studies two routes to high-quality German encoders under identical data and training constraints: 1) training from scratch and 2) converting decoders via LLM2Vec. We introduce two resources: ModernGBERT (134M, 1B), fully transparent German encoders in the ModernBERT style, and LLäMmleinVec (120M, 1B, 7B), decoder-to-encoder conversions trained with masked next-token prediction, both undergoing a context extension to 8.192 tokens. Across SuperGLEBer, ModernGBERT 1B sets a new state of the art (avg 0.808), surpassing GBERT Large (+4%) and the seven-times larger converted 7B model (0.787). On German MTEB after supervised fine-tuning, ModernGBERT 1B (0.551) approaches the converted 7B model (0.557). We release all models, checkpoints, datasets, and full training records, and introduce an encoder-adapted QA-NIAH evaluation. All in all, our results provide actionable guidance: when parameter efficiency and latency matter, from-scratch encoders dominate. When a pre-trained decoder exists and compute is a limited, conversion offers an effective alternative. ModernGBERT and LLäMmleinVec, including all code, data and intermediary checkpoints are published under a research-only RAIL license.

Figures (2)

• Figure 1: Performance on SuperGLEBer benchmark. $\bullet$ markers: encoders, $\blacktriangle$ markers: decoders. Dashed arrows: LLM2Vec conversion gains. Models of the same family are colored the same.
• Figure 2: Intermediate checkpoint evaluation. The solid black line shows the mean of six SuperGLEBer tasks (NLI, FactClaiming Comments, DB Aspect, WebCAGe, EuroParl, PAWSX Similarity). The top figure shows ModernGBERT 1B across pre-training and two context extension phases, with box plots representing all 29 SuperGLEBer tasks. For simplicity, no significant improvements between checkpoint pairs are marked with brackets (Wilcoxon signed-rank test). All other pairs of box plots show significant improvements of at least p < 0.01. The bottom figure shows LLäMmlein 1B after LLM2Vec conversion, including the starting point in (green) and the average score after switching the mask (red). Checkpoints on ext2 are shown alone and merged with the last ext1 checkpoint.