Llama-Embed-Nemotron-8B: A Universal Text Embedding Model for Multilingual and Cross-Lingual Tasks

TL;DR

llama-embed-nemotron-8b targets universal, multilingual text embeddings by converting a Llama-3.1-8B base into a bidirectional encoder and training it with a two-stage, data-mix–driven regime totaling $16.1$ million <query, document> pairs. The model achieves state-of-the-art results on the MMTEB benchmark through a combination of contrastive learning with InfoNCE loss, instruction-aware prompting, diverse synthetic data generated by multiple open-weight LLMs, hard negative mining, and model merging of six diverse checkpoints. Key contributions include comprehensive ablations on loss formulations, SDG strategies, and data composition, plus a commitment to open-source weights and curated datasets to foster reproducibility. The work’s practical impact spans retrieval, classification, and semantic similarity across many languages, offering a robust, open, and configurable embedding solution for multi-domain applications.

Abstract

We introduce llama-embed-nemotron-8b, an open-weights text embedding model that achieves state-of-the-art performance on the Multilingual Massive Text Embedding Benchmark (MMTEB) leaderboard as of October 21, 2025. While recent models show strong performance, their training data or methodologies are often not fully disclosed. We aim to address this by developing a fully open-source model, publicly releasing its weights and detailed ablation studies, and planning to share the curated training datasets. Our model demonstrates superior performance across all major embedding tasks -- including retrieval, classification and semantic textual similarity (STS) -- and excels in challenging multilingual scenarios, such as low-resource languages and cross-lingual setups. This state-of-the-art performance is driven by a novel data mix of 16.1 million query-document pairs, split between 7.7 million samples from public datasets and 8.4 million synthetically generated examples from various open-weight LLMs. One of our key contributions is a detailed ablation study analyzing core design choices, including a comparison of contrastive loss implementations, an evaluation of synthetic data generation (SDG) strategies, and the impact of model merging. The llama-embed-nemotron-8b is an instruction-aware model, supporting user-defined instructions to enhance performance for specific use-cases. This combination of top-tier performance, broad applicability, and user-driven flexibility enables it to serve as a universal text embedding solution.