Arctic-Embed 2.0: Multilingual Retrieval Without Compromise

TL;DR

Arctic-Embed 2.0 addresses efficient, high-quality multilingual dense retrieval and the common English retrieval gap observed in multilingual models. It uses a three-stage training recipe (masked language modeling, contrastive pretraining, contrastive finetuning) with two MLM-based base encoders and Matryoshka Representation Learning to compress embeddings by 3×–4× while preserving accuracy. Benchmarks on MTEB, MIRACL, and CLEF show competitive English and multilingual retrieval, with strong performance under 256-dim truncation and parity with or superiority to open-source baselines. The study also investigates cross-lingual transfer and the English performance gap, finding data quality and finetuning calibration can override potential costs of multilingual pretraining. These findings provide practical guidance for scalable multilingual IR and highlight open questions about how best to leverage distant-language data for cross-lingual performance.

Abstract

This paper presents the training methodology of Arctic-Embed 2.0, a set of open-source text embedding models built for accurate and efficient multilingual retrieval. While prior works have suffered from degraded English retrieval quality, Arctic-Embed 2.0 delivers competitive retrieval quality on multilingual and English-only benchmarks, and supports Matryoshka Representation Learning (MRL) for efficient embedding storage with significantly lower compressed quality degradation compared to alternatives. We detail the design and implementation, presenting several important open research questions that arose during model development. We conduct experiments exploring these research questions and include extensive discussion aimed at fostering further discussion in this field.

Figures (5)

• Figure 1: Single-vector dense retrieval performance of open-source multilingual embedding models with fewer than 1B parameters. Scores are average nDCG@10 on MTEB Retrieval muennighoff2023mteb and the subset of CLEF clef covering English, French, Spanish, Italian, and German.
• Figure 2: Hard-negative mining ablation studies. A stronger teacher embedding model and well-tuned false-positive cutoff led to improved downstream performance, while a random order of examples performed just as well as various approaches to creating easy-to-hard curricula.
• Figure 3: MIRACL performance (in nDCG@10) at different points during contrastive pretraining. Languages are grouped by linguistic families provided by zhang_miracl_2023. Dotted lines represent non-finetuned runs, while solid lines represent finetuned runs. The relative improvement or deterioration of model performance at the end (130K steps) compared to the 8K-step checkpoint is reported for both non-finetuned and finetuned runs.
• Figure 4: The impact of adding equal amounts of English, German, Spanish, or Chinese data to the existing English pretraining baseline ("en") on downstream retrieval performance. For Chinese data, error bars indicate the standard deviation across consistency filtering levels (top-{1, 5, 10, 20, 30} out of 3M), reflecting the effect of varying data quality.
• Figure 5: Breakdown of 1.41B contrastive pretraining samples by data source (top) and language (bottom).