What Matters When Building Universal Multilingual Named Entity Recognition Models?

TL;DR

This work tackles the challenge of building universal multilingual NER by systematically ablating four design dimensions—architecture (cross- vs bi-encoder), transformer backbone, loss function, and multilingual training data—across a controlled set of benchmarks. It finds that backbone-architecture compatibility and multilingual data diversity drive substantial gains, with binary cross-entropy often outperforming more complex losses. Based on these insights, the Otter model is trained to cover over 100 languages and achieves competitive results with much larger generative models, while maintaining efficiency. The study emphasizes the critical roles of per-language thresholding and diverse training data for robust cross-lingual NER performance, and provides public checkpoints and code to advance reproducibility.

Abstract

Recent progress in universal multilingual named entity recognition (NER) has been driven by advances in multilingual transformer models and task-specific architectures, loss functions, and training datasets. Despite substantial prior work, we find that many critical design decisions for such models are made without systematic justification, with architectural components, training objectives, and data sources evaluated only in combination rather than in isolation. We argue that these decisions impede progress in the field by making it difficult to identify which choices improve model performance. In this work, we conduct extensive experiments around architectures, transformer backbones, training objectives, and data composition across a wide range of languages. Based on these insights, we introduce Otter, a universal multilingual NER model supporting over 100 languages. Otter achieves consistent improvements over strong multilingual NER baselines, outperforming GLiNER-x-base by 5.3pp in F1 and achieves competitive performance compared to large generative models such as Qwen3-32B, while being substantially more efficient. We release model checkpoints, training and evaluation code to facilitate reproducibility and future research.

Figures (5)

• Figure 1: Overview of the design choices in universal multilingual NER systems. Prior work typically combines a choice of dataset, architecture, backbone, and training objective, only investigating their joint effects. In this work, we systematically combine these dimensions in a controlled setting to study their impact on performance and efficiency.
• Figure 2: Macro-averaged micro-F1 scores across all backbones evaluation benchmarks with different decision thresholds $t$. We observe that optimal performance is dependent on the transformer backbone.
• Figure 3: Micro F1, latency, and FLOPs as the number of labels increases.
• Figure 4: Example of a subword-boundary error. The prefix marker in eZimbabwe is labeled as O (red), while the following subwords are correctly predicted as ORG (green).
• Figure 5: The impact of using pre-tokenized text for training. We can use word boundaries to exclude spans from the loss resulting the higher positive-to-negative ratios.